{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "Congratulations for completing the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program!  In this notebook, you will learn how to control an agent in a more challenging environment, where the goal is to train a creature with four arms to walk forward.  **Note that this exercise is optional!**\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Crawler.app\"`\n",
    "- **Windows** (x86): `\"path/to/Crawler_Windows_x86/Crawler.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Crawler_Windows_x86_64/Crawler.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Crawler_Linux/Crawler.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Crawler_Linux/Crawler.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Crawler_Linux_NoVis/Crawler.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Crawler_Linux_NoVis/Crawler.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Crawler.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Crawler.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: CrawlerBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 129\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 20\n",
      "        Vector Action descriptions: , , , , , , , , , , , , , , , , , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='Crawler.exe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 12\n",
      "Size of each action: 20\n",
      "There are 12 agents. Each observes a state with length: 129\n",
      "The state for the first agent looks like: [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  2.25000000e+00\n",
      "  1.00000000e+00  0.00000000e+00  1.78813934e-07  0.00000000e+00\n",
      "  1.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  6.06093168e-01 -1.42857209e-01 -6.06078804e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  1.33339906e+00 -1.42857209e-01\n",
      " -1.33341408e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      " -6.06093347e-01 -1.42857209e-01 -6.06078625e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.33339953e+00 -1.42857209e-01\n",
      " -1.33341372e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      " -6.06093168e-01 -1.42857209e-01  6.06078804e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.33339906e+00 -1.42857209e-01\n",
      "  1.33341408e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  6.06093347e-01 -1.42857209e-01  6.06078625e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  1.33339953e+00 -1.42857209e-01\n",
      "  1.33341372e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "#states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "#scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "#while True:\n",
    "#    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "#    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "#    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "#    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "#    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "#    dones = env_info.local_done                        # see if episode finished\n",
    "#    scores += env_info.rewards                         # update the score (for each agent)\n",
    "#    states = next_states                               # roll over states to next time step\n",
    "#    if np.any(dones):                                  # exit loop if episode finished\n",
    "#        break\n",
    "#print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from  collections  import deque\n",
    "from itertools import count\n",
    "import torch\n",
    "import time\n",
    "from ppo_agent import Agent\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=8,\\\n",
    "              n_agent=num_agents, fc1_units=128, fc2_units=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1, Score: 48.61, Max: 48.61, Min: 48.61, T-max: 1000   \n",
      "Episode: 2, Score: 47.50, Max: 48.61, Min: 47.50, T-max: 1000   \n",
      "Episode: 3, Score: 45.66, Max: 48.61, Min: 45.66, T-max: 1000   \n",
      "Episode: 4, Score: 48.32, Max: 48.61, Min: 45.66, T-max: 1000   \n",
      "Episode: 5, Score: 49.08, Max: 49.08, Min: 45.66, T-max: 1000   \n",
      "Episode: 6, Score: 49.45, Max: 49.45, Min: 45.66, T-max: 1000   \n",
      "Episode: 7, Score: 49.53, Max: 49.53, Min: 45.66, T-max: 1000   \n",
      "Episode: 8, Score: 51.14, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 9, Score: 50.75, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 10, Score: 46.26, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "*** Episode 10 \t Average Score (over agents): 46.26 \t Average Score on 100 Episode: 48.63, Time: 00:02:11***\n",
      "Episode: 11, Score: 47.03, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 12, Score: 49.65, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 13, Score: 53.39, Max: 53.39, Min: 45.66, T-max: 1000   \n",
      "Episode: 14, Score: 53.76, Max: 53.76, Min: 45.66, T-max: 1000   \n",
      "Episode: 15, Score: 51.71, Max: 53.76, Min: 45.66, T-max: 1000   \n",
      "Episode: 16, Score: 54.96, Max: 54.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 17, Score: 54.30, Max: 54.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 18, Score: 53.48, Max: 54.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 19, Score: 53.13, Max: 54.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 20, Score: 57.29, Max: 57.29, Min: 45.66, T-max: 1000   \n",
      "*** Episode 20 \t Average Score (over agents): 57.29 \t Average Score on 100 Episode: 50.75, Time: 00:04:20***\n",
      "Episode: 21, Score: 52.79, Max: 57.29, Min: 45.66, T-max: 1000   \n",
      "Episode: 22, Score: 52.51, Max: 57.29, Min: 45.66, T-max: 1000   \n",
      "Episode: 23, Score: 57.20, Max: 57.29, Min: 45.66, T-max: 1000   \n",
      "Episode: 24, Score: 107.98, Max: 107.98, Min: 45.66, T-max: 1999   \n",
      "Episode: 25, Score: 58.27, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 26, Score: 55.97, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 27, Score: 55.51, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 28, Score: 55.87, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 29, Score: 56.41, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 30, Score: 57.10, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 30 \t Average Score (over agents): 57.10 \t Average Score on 100 Episode: 54.15, Time: 00:06:47***\n",
      "Episode: 31, Score: 60.13, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 32, Score: 56.94, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 33, Score: 52.80, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 34, Score: 64.32, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 35, Score: 59.08, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 36, Score: 52.60, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 37, Score: 60.61, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 38, Score: 59.86, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 39, Score: 56.63, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 40, Score: 56.11, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 40 \t Average Score (over agents): 56.11 \t Average Score on 100 Episode: 55.09, Time: 00:09:01***\n",
      "Episode: 41, Score: 58.01, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 42, Score: 61.22, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 43, Score: 62.21, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 44, Score: 62.52, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 45, Score: 61.94, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 46, Score: 58.17, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 47, Score: 63.81, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 48, Score: 62.80, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 49, Score: 67.10, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 50, Score: 66.93, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 50 \t Average Score (over agents): 66.93 \t Average Score on 100 Episode: 56.57, Time: 00:11:09***\n",
      "Episode: 51, Score: 63.05, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 52, Score: 65.90, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 53, Score: 66.75, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 54, Score: 67.92, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 55, Score: 69.89, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 56, Score: 69.84, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 57, Score: 69.13, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 58, Score: 66.15, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 59, Score: 68.83, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 60, Score: 65.87, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 60 \t Average Score (over agents): 65.87 \t Average Score on 100 Episode: 58.36, Time: 00:13:22***\n",
      "Episode: 61, Score: 68.35, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 62, Score: 66.47, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 63, Score: 66.83, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 64, Score: 75.38, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 65, Score: 66.64, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 66, Score: 59.91, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 67, Score: 71.04, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 68, Score: 76.62, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 69, Score: 72.07, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 70, Score: 68.02, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 70 \t Average Score (over agents): 68.02 \t Average Score on 100 Episode: 59.90, Time: 00:15:39***\n",
      "Episode: 71, Score: 74.09, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 72, Score: 73.02, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 73, Score: 78.71, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 74, Score: 56.06, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 75, Score: 70.54, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 76, Score: 71.66, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 77, Score: 71.02, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 78, Score: 77.74, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 79, Score: 80.36, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 80, Score: 69.26, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 80 \t Average Score (over agents): 69.26 \t Average Score on 100 Episode: 61.44, Time: 00:17:56***\n",
      "Episode: 81, Score: 77.52, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 82, Score: 80.57, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 83, Score: 71.07, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 84, Score: 77.34, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 85, Score: 74.20, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 86, Score: 65.32, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 87, Score: 74.80, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 88, Score: 77.05, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 89, Score: 78.59, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 90, Score: 79.43, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 90 \t Average Score (over agents): 79.43 \t Average Score on 100 Episode: 63.02, Time: 00:20:04***\n",
      "Episode: 91, Score: 90.58, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 92, Score: 83.91, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 93, Score: 87.73, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 94, Score: 90.11, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 95, Score: 81.61, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 96, Score: 83.91, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 97, Score: 102.21, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 98, Score: 87.14, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 99, Score: 93.58, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 100, Score: 95.18, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 100 \t Average Score (over agents): 95.18 \t Average Score on 100 Episode: 65.67, Time: 00:22:15***\n",
      "Episode: 101, Score: 87.57, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 102, Score: 95.06, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 103, Score: 99.46, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 104, Score: 85.37, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 105, Score: 88.30, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 106, Score: 93.26, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 107, Score: 86.22, Max: 107.98, Min: 45.66, T-max: 1000   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 108, Score: 86.39, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 109, Score: 92.95, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 110, Score: 95.46, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 110 \t Average Score (over agents): 95.46 \t Average Score on 100 Episode: 69.91, Time: 00:24:40***\n",
      "Episode: 111, Score: 96.57, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 112, Score: 107.68, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 113, Score: 103.98, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 114, Score: 105.25, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 115, Score: 125.42, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 116, Score: 98.82, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 117, Score: 94.93, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 118, Score: 91.78, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 119, Score: 102.73, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 120, Score: 97.88, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "*** Episode 120 \t Average Score (over agents): 97.88 \t Average Score on 100 Episode: 74.87, Time: 00:27:06***\n",
      "Episode: 121, Score: 109.84, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 122, Score: 99.40, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 123, Score: 93.42, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 124, Score: 106.46, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 125, Score: 102.88, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 126, Score: 105.21, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 127, Score: 112.26, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 128, Score: 102.49, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 129, Score: 118.29, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 130, Score: 132.80, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "*** Episode 130 \t Average Score (over agents): 132.80 \t Average Score on 100 Episode: 79.61, Time: 00:29:25***\n",
      "Episode: 131, Score: 125.35, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 132, Score: 123.34, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 133, Score: 117.92, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 134, Score: 118.95, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 135, Score: 124.56, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 136, Score: 110.89, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 137, Score: 118.94, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 138, Score: 131.62, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 139, Score: 136.17, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 140, Score: 129.26, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "*** Episode 140 \t Average Score (over agents): 129.26 \t Average Score on 100 Episode: 86.19, Time: 00:31:53***\n",
      "Episode: 141, Score: 128.87, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 142, Score: 118.09, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 143, Score: 136.13, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 144, Score: 116.34, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 145, Score: 142.87, Max: 142.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 146, Score: 136.71, Max: 142.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 147, Score: 135.22, Max: 142.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 148, Score: 134.39, Max: 142.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 149, Score: 146.28, Max: 146.28, Min: 45.66, T-max: 1000   \n",
      "Episode: 150, Score: 136.92, Max: 146.28, Min: 45.66, T-max: 1000   \n",
      "*** Episode 150 \t Average Score (over agents): 136.92 \t Average Score on 100 Episode: 93.26, Time: 00:34:13***\n",
      "Episode: 151, Score: 146.52, Max: 146.52, Min: 45.66, T-max: 1000   \n",
      "Episode: 152, Score: 153.44, Max: 153.44, Min: 45.66, T-max: 1000   \n",
      "Episode: 153, Score: 145.44, Max: 153.44, Min: 45.66, T-max: 1000   \n",
      "Episode: 154, Score: 155.97, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 155, Score: 146.66, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 156, Score: 141.29, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 157, Score: 151.83, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 158, Score: 145.71, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 159, Score: 134.30, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 160, Score: 145.89, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "*** Episode 160 \t Average Score (over agents): 145.89 \t Average Score on 100 Episode: 101.20, Time: 00:36:30***\n",
      "Episode: 161, Score: 151.97, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 162, Score: 144.06, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 163, Score: 141.81, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 164, Score: 159.43, Max: 159.43, Min: 45.66, T-max: 1000   \n",
      "Episode: 165, Score: 142.53, Max: 159.43, Min: 45.66, T-max: 1000   \n",
      "Episode: 166, Score: 165.84, Max: 165.84, Min: 45.66, T-max: 1000   \n",
      "Episode: 167, Score: 151.11, Max: 165.84, Min: 45.66, T-max: 1000   \n",
      "Episode: 168, Score: 156.89, Max: 165.84, Min: 45.66, T-max: 1000   \n",
      "Episode: 169, Score: 157.89, Max: 165.84, Min: 45.66, T-max: 1000   \n",
      "Episode: 170, Score: 171.14, Max: 171.14, Min: 45.66, T-max: 1000   \n",
      "*** Episode 170 \t Average Score (over agents): 171.14 \t Average Score on 100 Episode: 109.71, Time: 00:38:50***\n",
      "Episode: 171, Score: 167.94, Max: 171.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 172, Score: 153.36, Max: 171.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 173, Score: 163.89, Max: 171.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 174, Score: 178.58, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 175, Score: 166.16, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 176, Score: 147.46, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 177, Score: 175.76, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 178, Score: 171.11, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 179, Score: 159.70, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 180, Score: 165.96, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "*** Episode 180 \t Average Score (over agents): 165.96 \t Average Score on 100 Episode: 118.98, Time: 00:41:05***\n",
      "Episode: 181, Score: 175.61, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 182, Score: 181.50, Max: 181.50, Min: 45.66, T-max: 1000   \n",
      "Episode: 183, Score: 181.38, Max: 181.50, Min: 45.66, T-max: 1000   \n",
      "Episode: 184, Score: 189.66, Max: 189.66, Min: 45.66, T-max: 1000   \n",
      "Episode: 185, Score: 163.22, Max: 189.66, Min: 45.66, T-max: 1000   \n",
      "Episode: 186, Score: 170.11, Max: 189.66, Min: 45.66, T-max: 1000   \n",
      "Episode: 187, Score: 197.68, Max: 197.68, Min: 45.66, T-max: 1000   \n",
      "Episode: 188, Score: 203.09, Max: 203.09, Min: 45.66, T-max: 1000   \n",
      "Episode: 189, Score: 194.66, Max: 203.09, Min: 45.66, T-max: 1000   \n",
      "Episode: 190, Score: 171.05, Max: 203.09, Min: 45.66, T-max: 1000   \n",
      "*** Episode 190 \t Average Score (over agents): 171.05 \t Average Score on 100 Episode: 129.71, Time: 00:43:19***\n",
      "Episode: 191, Score: 204.88, Max: 204.88, Min: 45.66, T-max: 1000   \n",
      "Episode: 192, Score: 365.82, Max: 365.82, Min: 45.66, T-max: 1999   \n",
      "Episode: 193, Score: 207.89, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 194, Score: 210.45, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 195, Score: 187.19, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 196, Score: 196.92, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 197, Score: 192.01, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 198, Score: 191.07, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 199, Score: 217.43, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 200, Score: 218.17, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "*** Episode 200 \t Average Score (over agents): 218.17 \t Average Score on 100 Episode: 142.66, Time: 00:45:51***\n",
      "Episode: 201, Score: 209.03, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 202, Score: 202.94, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 203, Score: 207.35, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 204, Score: 208.74, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 205, Score: 217.97, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 206, Score: 196.01, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 207, Score: 224.49, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 208, Score: 211.50, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 209, Score: 220.33, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 210, Score: 218.93, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "*** Episode 210 \t Average Score (over agents): 218.93 \t Average Score on 100 Episode: 154.74, Time: 00:48:07***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 211, Score: 206.65, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 212, Score: 207.38, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 213, Score: 223.16, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 214, Score: 236.69, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 215, Score: 233.84, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 216, Score: 235.96, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 217, Score: 226.66, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 218, Score: 238.24, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 219, Score: 211.15, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 220, Score: 217.61, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "*** Episode 220 \t Average Score (over agents): 217.61 \t Average Score on 100 Episode: 166.86, Time: 00:50:19***\n",
      "Episode: 221, Score: 231.79, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 222, Score: 239.67, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 223, Score: 240.89, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 224, Score: 248.11, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 225, Score: 261.12, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 226, Score: 512.83, Max: 512.83, Min: 45.66, T-max: 1999   \n",
      "Episode: 227, Score: 243.24, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 228, Score: 272.20, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 229, Score: 257.68, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 230, Score: 278.05, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "*** Episode 230 \t Average Score (over agents): 278.05 \t Average Score on 100 Episode: 183.88, Time: 00:52:49***\n",
      "Episode: 231, Score: 284.37, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 232, Score: 275.33, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 233, Score: 244.79, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 234, Score: 260.68, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 235, Score: 259.12, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 236, Score: 264.47, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 237, Score: 281.34, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 238, Score: 299.17, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 239, Score: 291.71, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 240, Score: 289.92, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "*** Episode 240 \t Average Score (over agents): 289.92 \t Average Score on 100 Episode: 199.02, Time: 00:55:05***\n",
      "Episode: 241, Score: 247.10, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 242, Score: 288.12, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 243, Score: 496.16, Max: 512.83, Min: 45.66, T-max: 1999   \n",
      "Episode: 244, Score: 245.71, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 245, Score: 519.05, Max: 519.05, Min: 45.66, T-max: 1999   \n",
      "Episode: 246, Score: 281.62, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 247, Score: 267.02, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 248, Score: 271.96, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 249, Score: 288.12, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 250, Score: 254.84, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "*** Episode 250 \t Average Score (over agents): 254.84 \t Average Score on 100 Episode: 217.30, Time: 00:57:37***\n",
      "Episode: 251, Score: 278.21, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 252, Score: 271.08, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 253, Score: 267.94, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 254, Score: 280.15, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 255, Score: 260.06, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 256, Score: 313.75, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 257, Score: 284.41, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 258, Score: 279.29, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 259, Score: 599.58, Max: 599.58, Min: 45.66, T-max: 1999   \n",
      "Episode: 260, Score: 284.67, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "*** Episode 260 \t Average Score (over agents): 284.67 \t Average Score on 100 Episode: 233.82, Time: 00:59:56***\n",
      "Episode: 261, Score: 554.62, Max: 599.58, Min: 45.66, T-max: 1999   \n",
      "Episode: 262, Score: 304.01, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 263, Score: 284.20, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 264, Score: 305.12, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 265, Score: 537.70, Max: 599.58, Min: 45.66, T-max: 1999   \n",
      "Episode: 266, Score: 300.89, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 267, Score: 614.41, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 268, Score: 265.37, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 269, Score: 288.68, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 270, Score: 557.26, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "*** Episode 270 \t Average Score (over agents): 557.26 \t Average Score on 100 Episode: 258.52, Time: 01:02:53***\n",
      "Episode: 271, Score: 581.81, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 272, Score: 285.30, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 273, Score: 243.16, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 274, Score: 565.17, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 275, Score: 309.74, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 276, Score: 275.60, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 277, Score: 578.01, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 278, Score: 299.50, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 279, Score: 309.69, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 280, Score: 299.01, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "*** Episode 280 \t Average Score (over agents): 299.01 \t Average Score on 100 Episode: 279.49, Time: 01:05:37***\n",
      "Episode: 281, Score: 281.49, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 282, Score: 602.05, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 283, Score: 317.24, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 284, Score: 628.51, Max: 628.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 285, Score: 293.56, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 286, Score: 278.92, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 287, Score: 289.26, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 288, Score: 292.93, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 289, Score: 303.01, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 290, Score: 285.28, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "*** Episode 290 \t Average Score (over agents): 285.28 \t Average Score on 100 Episode: 296.93, Time: 01:08:07***\n",
      "Episode: 291, Score: 288.36, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 292, Score: 271.69, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 293, Score: 290.97, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 294, Score: 304.28, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 295, Score: 286.75, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 296, Score: 290.81, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 297, Score: 321.01, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 298, Score: 296.45, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 299, Score: 302.02, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 300, Score: 331.31, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "*** Episode 300 \t Average Score (over agents): 331.31 \t Average Score on 100 Episode: 304.85, Time: 01:10:13***\n",
      "Episode: 301, Score: 310.42, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 302, Score: 299.09, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 303, Score: 314.84, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 304, Score: 606.47, Max: 628.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 305, Score: 343.87, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 306, Score: 615.90, Max: 628.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 307, Score: 315.54, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 308, Score: 625.97, Max: 628.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 309, Score: 322.68, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 310, Score: 308.76, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "*** Episode 310 \t Average Score (over agents): 308.76 \t Average Score on 100 Episode: 324.31, Time: 01:12:56***\n",
      "Episode: 311, Score: 329.49, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 312, Score: 337.64, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 313, Score: 337.85, Max: 628.51, Min: 45.66, T-max: 1000   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 314, Score: 361.98, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 315, Score: 721.77, Max: 721.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 316, Score: 336.87, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 317, Score: 352.17, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 318, Score: 356.57, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 319, Score: 341.65, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 320, Score: 347.49, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "*** Episode 320 \t Average Score (over agents): 347.49 \t Average Score on 100 Episode: 340.18, Time: 01:15:16***\n",
      "Episode: 321, Score: 381.17, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 322, Score: 674.16, Max: 721.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 323, Score: 363.54, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 324, Score: 352.78, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 325, Score: 369.42, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 326, Score: 341.48, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 327, Score: 364.45, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 328, Score: 353.11, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 329, Score: 385.92, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 330, Score: 723.96, Max: 723.96, Min: 45.66, T-max: 1999   \n",
      "*** Episode 330 \t Average Score (over agents): 723.96 \t Average Score on 100 Episode: 355.42, Time: 01:18:01***\n",
      "Episode: 331, Score: 373.99, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 332, Score: 371.51, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 333, Score: 362.85, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 334, Score: 357.82, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 335, Score: 374.23, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 336, Score: 358.56, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 337, Score: 802.56, Max: 802.56, Min: 45.66, T-max: 1999   \n",
      "Episode: 338, Score: 358.88, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 339, Score: 380.17, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 340, Score: 395.11, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "*** Episode 340 \t Average Score (over agents): 395.11 \t Average Score on 100 Episode: 369.27, Time: 01:20:29***\n",
      "Episode: 341, Score: 371.06, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 342, Score: 392.76, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 343, Score: 402.43, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 344, Score: 374.99, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 345, Score: 421.97, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 346, Score: 792.96, Max: 802.56, Min: 45.66, T-max: 1999   \n",
      "Episode: 347, Score: 739.93, Max: 802.56, Min: 45.66, T-max: 1999   \n",
      "Episode: 348, Score: 400.26, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 349, Score: 853.10, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "Episode: 350, Score: 844.46, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "*** Episode 350 \t Average Score (over agents): 844.46 \t Average Score on 100 Episode: 393.61, Time: 01:23:41***\n",
      "Episode: 351, Score: 394.59, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 352, Score: 428.32, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 353, Score: 803.14, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "Episode: 354, Score: 442.46, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 355, Score: 813.76, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "Episode: 356, Score: 413.05, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 357, Score: 440.39, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 358, Score: 426.65, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 359, Score: 435.81, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 360, Score: 408.87, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "*** Episode 360 \t Average Score (over agents): 408.87 \t Average Score on 100 Episode: 412.49, Time: 01:26:23***\n",
      "Episode: 361, Score: 432.91, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 362, Score: 425.07, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 363, Score: 429.68, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 364, Score: 438.73, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 365, Score: 423.69, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 366, Score: 432.64, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 367, Score: 460.11, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 368, Score: 448.60, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 369, Score: 411.76, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 370, Score: 840.25, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "*** Episode 370 \t Average Score (over agents): 840.25 \t Average Score on 100 Episode: 419.80, Time: 01:28:54***\n",
      "Episode: 371, Score: 442.79, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 372, Score: 438.56, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 373, Score: 448.86, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 374, Score: 475.87, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 375, Score: 434.69, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 376, Score: 451.37, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 377, Score: 458.01, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 378, Score: 441.46, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 379, Score: 889.46, Max: 889.46, Min: 45.66, T-max: 1999   \n",
      "Episode: 380, Score: 898.63, Max: 898.63, Min: 45.66, T-max: 1999   \n",
      "*** Episode 380 \t Average Score (over agents): 898.63 \t Average Score on 100 Episode: 436.13, Time: 01:31:36***\n",
      "Episode: 381, Score: 464.63, Max: 898.63, Min: 45.66, T-max: 1000   \n",
      "Episode: 382, Score: 470.39, Max: 898.63, Min: 45.66, T-max: 1000   \n",
      "Episode: 383, Score: 854.28, Max: 898.63, Min: 45.66, T-max: 1999   \n",
      "Episode: 384, Score: 871.41, Max: 898.63, Min: 45.66, T-max: 1999   \n",
      "Episode: 385, Score: 414.61, Max: 898.63, Min: 45.66, T-max: 1000   \n",
      "Episode: 386, Score: 910.62, Max: 910.62, Min: 45.66, T-max: 1999   \n",
      "Episode: 387, Score: 445.40, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 388, Score: 462.56, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 389, Score: 435.20, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 390, Score: 442.32, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "*** Episode 390 \t Average Score (over agents): 442.32 \t Average Score on 100 Episode: 458.12, Time: 01:34:24***\n",
      "Episode: 391, Score: 470.94, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 392, Score: 447.15, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 393, Score: 449.33, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 394, Score: 456.29, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 395, Score: 469.65, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 396, Score: 467.13, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 397, Score: 441.31, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 398, Score: 936.13, Max: 936.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 399, Score: 468.85, Max: 936.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 400, Score: 482.14, Max: 936.13, Min: 45.66, T-max: 1000   \n",
      "*** Episode 400 \t Average Score (over agents): 482.14 \t Average Score on 100 Episode: 479.17, Time: 01:36:41***\n",
      "Episode: 401, Score: 927.88, Max: 936.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 402, Score: 473.94, Max: 936.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 403, Score: 448.12, Max: 936.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 404, Score: 934.24, Max: 936.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 405, Score: 940.13, Max: 940.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 406, Score: 925.06, Max: 940.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 407, Score: 919.30, Max: 940.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 408, Score: 470.64, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 409, Score: 420.46, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 410, Score: 480.00, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "*** Episode 410 \t Average Score (over agents): 480.00 \t Average Score on 100 Episode: 507.93, Time: 01:39:49***\n",
      "Episode: 411, Score: 496.79, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 412, Score: 492.97, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 413, Score: 476.59, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 414, Score: 947.74, Max: 947.74, Min: 45.66, T-max: 1999   \n",
      "Episode: 415, Score: 987.41, Max: 987.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 416, Score: 440.42, Max: 987.41, Min: 45.66, T-max: 1000   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 417, Score: 481.76, Max: 987.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 418, Score: 467.07, Max: 987.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 419, Score: 501.55, Max: 987.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 420, Score: 496.19, Max: 987.41, Min: 45.66, T-max: 1000   \n",
      "*** Episode 420 \t Average Score (over agents): 496.19 \t Average Score on 100 Episode: 527.58, Time: 01:42:21***\n",
      "Episode: 421, Score: 982.41, Max: 987.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 422, Score: 998.00, Max: 998.00, Min: 45.66, T-max: 1999   \n",
      "Episode: 423, Score: 494.59, Max: 998.00, Min: 45.66, T-max: 1000   \n",
      "Episode: 424, Score: 506.94, Max: 998.00, Min: 45.66, T-max: 1000   \n",
      "Episode: 425, Score: 514.57, Max: 998.00, Min: 45.66, T-max: 1000   \n",
      "Episode: 426, Score: 485.72, Max: 998.00, Min: 45.66, T-max: 1000   \n",
      "Episode: 427, Score: 1055.72, Max: 1055.72, Min: 45.66, T-max: 1999   \n",
      "Episode: 428, Score: 525.59, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 429, Score: 517.97, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 430, Score: 956.56, Max: 1055.72, Min: 45.66, T-max: 1999   \n",
      "*** Episode 430 \t Average Score (over agents): 956.56 \t Average Score on 100 Episode: 554.86, Time: 01:45:17***\n",
      "Episode: 431, Score: 534.23, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 432, Score: 520.05, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 433, Score: 537.12, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 434, Score: 516.69, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 435, Score: 533.51, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 436, Score: 484.10, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 437, Score: 505.07, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 438, Score: 512.75, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 439, Score: 532.26, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 440, Score: 983.60, Max: 1055.72, Min: 45.66, T-max: 1999   \n",
      "*** Episode 440 \t Average Score (over agents): 983.60 \t Average Score on 100 Episode: 570.10, Time: 01:47:34***\n",
      "Episode: 441, Score: 1067.37, Max: 1067.37, Min: 45.66, T-max: 1999   \n",
      "Episode: 442, Score: 541.69, Max: 1067.37, Min: 45.66, T-max: 1000   \n",
      "Episode: 443, Score: 1074.77, Max: 1074.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 444, Score: 504.74, Max: 1074.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 445, Score: 996.96, Max: 1074.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 446, Score: 1031.91, Max: 1074.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 447, Score: 1056.65, Max: 1074.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 448, Score: 1113.40, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 449, Score: 1079.28, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 450, Score: 1068.86, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "*** Episode 450 \t Average Score (over agents): 1068.86 \t Average Score on 100 Episode: 609.52, Time: 01:51:20***\n",
      "Episode: 451, Score: 1046.80, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 452, Score: 535.06, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 453, Score: 563.16, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 454, Score: 557.11, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 455, Score: 504.79, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 456, Score: 532.65, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 457, Score: 535.53, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 458, Score: 1089.69, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 459, Score: 1072.80, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 460, Score: 1125.87, Max: 1125.87, Min: 45.66, T-max: 1999   \n",
      "*** Episode 460 \t Average Score (over agents): 1125.87 \t Average Score on 100 Episode: 635.08, Time: 01:54:16***\n",
      "Episode: 461, Score: 559.03, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 462, Score: 567.15, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 463, Score: 522.76, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 464, Score: 499.29, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 465, Score: 1099.18, Max: 1125.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 466, Score: 555.60, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 467, Score: 1108.91, Max: 1125.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 468, Score: 580.50, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 469, Score: 541.42, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 470, Score: 572.34, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "*** Episode 470 \t Average Score (over agents): 572.34 \t Average Score on 100 Episode: 653.71, Time: 01:56:46***\n",
      "Episode: 471, Score: 1158.59, Max: 1158.59, Min: 45.66, T-max: 1999   \n",
      "Episode: 472, Score: 592.24, Max: 1158.59, Min: 45.66, T-max: 1000   \n",
      "Episode: 473, Score: 1122.65, Max: 1158.59, Min: 45.66, T-max: 1999   \n",
      "Episode: 474, Score: 574.30, Max: 1158.59, Min: 45.66, T-max: 1000   \n",
      "Episode: 475, Score: 1149.06, Max: 1158.59, Min: 45.66, T-max: 1999   \n",
      "Episode: 476, Score: 600.69, Max: 1158.59, Min: 45.66, T-max: 1000   \n",
      "Episode: 477, Score: 1163.77, Max: 1163.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 478, Score: 571.27, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 479, Score: 1121.20, Max: 1163.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 480, Score: 574.24, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "*** Episode 480 \t Average Score (over agents): 574.24 \t Average Score on 100 Episode: 686.19, Time: 01:59:55***\n",
      "Episode: 481, Score: 585.58, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 482, Score: 1141.43, Max: 1163.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 483, Score: 566.41, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 484, Score: 564.55, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 485, Score: 599.55, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 486, Score: 1081.23, Max: 1163.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 487, Score: 607.20, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 488, Score: 1189.23, Max: 1189.23, Min: 45.66, T-max: 1999   \n",
      "Episode: 489, Score: 1149.55, Max: 1189.23, Min: 45.66, T-max: 1999   \n",
      "Episode: 490, Score: 1090.40, Max: 1189.23, Min: 45.66, T-max: 1999   \n",
      "*** Episode 490 \t Average Score (over agents): 1090.40 \t Average Score on 100 Episode: 714.23, Time: 02:03:03***\n",
      "Episode: 491, Score: 571.54, Max: 1189.23, Min: 45.66, T-max: 1000   \n",
      "Episode: 492, Score: 608.06, Max: 1189.23, Min: 45.66, T-max: 1000   \n",
      "Episode: 493, Score: 570.98, Max: 1189.23, Min: 45.66, T-max: 1000   \n",
      "Episode: 494, Score: 1196.87, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 495, Score: 557.74, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 496, Score: 594.90, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 497, Score: 1136.21, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 498, Score: 1136.56, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 499, Score: 618.71, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 500, Score: 1184.71, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "*** Episode 500 \t Average Score (over agents): 1184.71 \t Average Score on 100 Episode: 745.10, Time: 02:05:58***\n",
      "Episode: 501, Score: 586.75, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 502, Score: 557.24, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 503, Score: 1070.01, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 504, Score: 553.20, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 505, Score: 1155.20, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 506, Score: 601.44, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 507, Score: 557.03, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 508, Score: 1193.86, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 509, Score: 1225.06, Max: 1225.06, Min: 45.66, T-max: 1999   \n",
      "Episode: 510, Score: 615.54, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "*** Episode 510 \t Average Score (over agents): 615.54 \t Average Score on 100 Episode: 756.86, Time: 02:08:56***\n",
      "Episode: 511, Score: 557.27, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "Episode: 512, Score: 579.10, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "Episode: 513, Score: 565.95, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "Episode: 514, Score: 634.08, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "Episode: 515, Score: 1245.49, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 516, Score: 1179.22, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 517, Score: 540.43, Max: 1245.49, Min: 45.66, T-max: 1000   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 518, Score: 1203.06, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 519, Score: 1207.55, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 520, Score: 620.01, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "*** Episode 520 \t Average Score (over agents): 620.01 \t Average Score on 100 Episode: 782.30, Time: 02:11:52***\n",
      "Episode: 521, Score: 599.54, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 522, Score: 1189.25, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 523, Score: 576.92, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 524, Score: 591.79, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 525, Score: 570.08, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 526, Score: 620.79, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 527, Score: 628.96, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 528, Score: 594.24, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 529, Score: 1215.60, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 530, Score: 1228.80, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "*** Episode 530 \t Average Score (over agents): 1228.80 \t Average Score on 100 Episode: 790.08, Time: 02:14:36***\n",
      "Episode: 531, Score: 1189.87, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 532, Score: 1196.51, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 533, Score: 612.32, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 534, Score: 611.27, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 535, Score: 640.34, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 536, Score: 614.38, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 537, Score: 1105.10, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 538, Score: 1267.72, Max: 1267.72, Min: 45.66, T-max: 1999   \n",
      "Episode: 539, Score: 1226.69, Max: 1267.72, Min: 45.66, T-max: 1999   \n",
      "Episode: 540, Score: 605.05, Max: 1267.72, Min: 45.66, T-max: 1000   \n",
      "*** Episode 540 \t Average Score (over agents): 605.05 \t Average Score on 100 Episode: 824.17, Time: 02:17:43***\n",
      "Episode: 541, Score: 623.47, Max: 1267.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 542, Score: 635.94, Max: 1267.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 543, Score: 625.11, Max: 1267.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 544, Score: 1177.38, Max: 1267.72, Min: 45.66, T-max: 1999   \n",
      "Episode: 545, Score: 1263.96, Max: 1267.72, Min: 45.66, T-max: 1999   \n",
      "Episode: 546, Score: 1254.99, Max: 1267.72, Min: 45.66, T-max: 1999   \n",
      "Episode: 547, Score: 1267.13, Max: 1267.72, Min: 45.66, T-max: 1999   \n",
      "Episode: 548, Score: 630.68, Max: 1267.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 549, Score: 580.99, Max: 1267.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 550, Score: 1254.06, Max: 1267.72, Min: 45.66, T-max: 1999   \n",
      "*** Episode 550 \t Average Score (over agents): 1254.06 \t Average Score on 100 Episode: 821.95, Time: 02:20:52***\n",
      "Episode: 551, Score: 1228.65, Max: 1267.72, Min: 45.66, T-max: 1999   \n",
      "Episode: 552, Score: 1299.75, Max: 1299.75, Min: 45.66, T-max: 1999   \n",
      "Episode: 553, Score: 660.66, Max: 1299.75, Min: 45.66, T-max: 1000   \n",
      "Episode: 554, Score: 620.92, Max: 1299.75, Min: 45.66, T-max: 1000   \n",
      "Episode: 555, Score: 649.60, Max: 1299.75, Min: 45.66, T-max: 1000   \n",
      "Episode: 556, Score: 1326.45, Max: 1326.45, Min: 45.66, T-max: 1999   \n",
      "Episode: 557, Score: 647.47, Max: 1326.45, Min: 45.66, T-max: 1000   \n",
      "Episode: 558, Score: 672.47, Max: 1326.45, Min: 45.66, T-max: 1000   \n",
      "Episode: 559, Score: 1300.68, Max: 1326.45, Min: 45.66, T-max: 1999   \n",
      "Episode: 560, Score: 1189.64, Max: 1326.45, Min: 45.66, T-max: 1999   \n",
      "*** Episode 560 \t Average Score (over agents): 1189.64 \t Average Score on 100 Episode: 842.28, Time: 02:23:58***\n",
      "Episode: 561, Score: 612.39, Max: 1326.45, Min: 45.66, T-max: 1000   \n",
      "Episode: 562, Score: 681.29, Max: 1326.45, Min: 45.66, T-max: 1000   \n",
      "Episode: 563, Score: 682.65, Max: 1326.45, Min: 45.66, T-max: 1000   \n",
      "Episode: 564, Score: 1306.48, Max: 1326.45, Min: 45.66, T-max: 1999   \n",
      "Episode: 565, Score: 664.28, Max: 1326.45, Min: 45.66, T-max: 1000   \n",
      "Episode: 566, Score: 1341.29, Max: 1341.29, Min: 45.66, T-max: 1999   \n",
      "Episode: 567, Score: 617.22, Max: 1341.29, Min: 45.66, T-max: 1000   \n",
      "Episode: 568, Score: 665.54, Max: 1341.29, Min: 45.66, T-max: 1000   \n",
      "Episode: 569, Score: 1342.28, Max: 1342.28, Min: 45.66, T-max: 1999   \n",
      "Episode: 570, Score: 682.84, Max: 1342.28, Min: 45.66, T-max: 1000   \n",
      "*** Episode 570 \t Average Score (over agents): 682.84 \t Average Score on 100 Episode: 862.18, Time: 02:26:42***\n",
      "Episode: 571, Score: 1306.38, Max: 1342.28, Min: 45.66, T-max: 1999   \n",
      "Episode: 572, Score: 649.62, Max: 1342.28, Min: 45.66, T-max: 1000   \n",
      "Episode: 573, Score: 1338.44, Max: 1342.28, Min: 45.66, T-max: 1999   \n",
      "Episode: 574, Score: 674.46, Max: 1342.28, Min: 45.66, T-max: 1000   \n",
      "Episode: 575, Score: 659.91, Max: 1342.28, Min: 45.66, T-max: 1000   \n",
      "Episode: 576, Score: 665.93, Max: 1342.28, Min: 45.66, T-max: 1000   \n",
      "Episode: 577, Score: 664.54, Max: 1342.28, Min: 45.66, T-max: 1000   \n",
      "Episode: 578, Score: 676.27, Max: 1342.28, Min: 45.66, T-max: 1000   \n",
      "Episode: 579, Score: 1355.44, Max: 1355.44, Min: 45.66, T-max: 1999   \n",
      "Episode: 580, Score: 1369.32, Max: 1369.32, Min: 45.66, T-max: 1999   \n",
      "*** Episode 580 \t Average Score (over agents): 1369.32 \t Average Score on 100 Episode: 869.51, Time: 02:29:37***\n",
      "Episode: 581, Score: 1358.59, Max: 1369.32, Min: 45.66, T-max: 1999   \n",
      "Episode: 582, Score: 681.51, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "Episode: 583, Score: 658.15, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "Episode: 584, Score: 692.64, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "Episode: 585, Score: 694.18, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "Episode: 586, Score: 663.18, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "Episode: 587, Score: 686.91, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "Episode: 588, Score: 695.76, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "Episode: 589, Score: 1366.91, Max: 1369.32, Min: 45.66, T-max: 1999   \n",
      "Episode: 590, Score: 704.77, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "*** Episode 590 \t Average Score (over agents): 704.77 \t Average Score on 100 Episode: 865.78, Time: 02:32:08***\n",
      "Episode: 591, Score: 691.49, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "Episode: 592, Score: 670.49, Max: 1369.32, Min: 45.66, T-max: 1000   \n",
      "Episode: 593, Score: 1267.74, Max: 1369.32, Min: 45.66, T-max: 1999   \n",
      "Episode: 594, Score: 1357.95, Max: 1369.32, Min: 45.66, T-max: 1999   \n",
      "Episode: 595, Score: 1372.99, Max: 1372.99, Min: 45.66, T-max: 1999   \n",
      "Episode: 596, Score: 1407.90, Max: 1407.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 597, Score: 1390.86, Max: 1407.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 598, Score: 1392.84, Max: 1407.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 599, Score: 695.04, Max: 1407.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 600, Score: 709.00, Max: 1407.90, Min: 45.66, T-max: 1000   \n",
      "*** Episode 600 \t Average Score (over agents): 709.00 \t Average Score on 100 Episode: 893.58, Time: 02:35:28***\n",
      "Episode: 601, Score: 690.72, Max: 1407.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 602, Score: 715.68, Max: 1407.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 603, Score: 693.91, Max: 1407.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 604, Score: 707.69, Max: 1407.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 605, Score: 660.08, Max: 1407.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 606, Score: 704.00, Max: 1407.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 607, Score: 705.87, Max: 1407.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 608, Score: 1403.96, Max: 1407.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 609, Score: 1409.43, Max: 1409.43, Min: 45.66, T-max: 1999   \n",
      "Episode: 610, Score: 1461.16, Max: 1461.16, Min: 45.66, T-max: 1999   \n",
      "*** Episode 610 \t Average Score (over agents): 1461.16 \t Average Score on 100 Episode: 903.95, Time: 02:38:11***\n",
      "Episode: 611, Score: 1413.19, Max: 1461.16, Min: 45.66, T-max: 1999   \n",
      "Episode: 612, Score: 1478.36, Max: 1478.36, Min: 45.66, T-max: 1999   \n",
      "Episode: 613, Score: 1428.50, Max: 1478.36, Min: 45.66, T-max: 1999   \n",
      "Episode: 614, Score: 727.47, Max: 1478.36, Min: 45.66, T-max: 1000   \n",
      "Episode: 615, Score: 1414.23, Max: 1478.36, Min: 45.66, T-max: 1999   \n",
      "Episode: 616, Score: 1464.33, Max: 1478.36, Min: 45.66, T-max: 1999   \n",
      "Episode: 617, Score: 1404.02, Max: 1478.36, Min: 45.66, T-max: 1999   \n",
      "Episode: 618, Score: 1411.32, Max: 1478.36, Min: 45.66, T-max: 1999   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 619, Score: 1466.80, Max: 1478.36, Min: 45.66, T-max: 1999   \n",
      "Episode: 620, Score: 1483.78, Max: 1483.78, Min: 45.66, T-max: 1999   \n",
      "*** Episode 620 \t Average Score (over agents): 1483.78 \t Average Score on 100 Episode: 957.55, Time: 02:42:09***\n",
      "Episode: 621, Score: 1465.28, Max: 1483.78, Min: 45.66, T-max: 1999   \n",
      "Episode: 622, Score: 751.92, Max: 1483.78, Min: 45.66, T-max: 1000   \n",
      "Episode: 623, Score: 708.25, Max: 1483.78, Min: 45.66, T-max: 1000   \n",
      "Episode: 624, Score: 735.05, Max: 1483.78, Min: 45.66, T-max: 1000   \n",
      "Episode: 625, Score: 1491.93, Max: 1491.93, Min: 45.66, T-max: 1999   \n",
      "Episode: 626, Score: 1490.34, Max: 1491.93, Min: 45.66, T-max: 1999   \n",
      "Episode: 627, Score: 755.81, Max: 1491.93, Min: 45.66, T-max: 1000   \n",
      "Episode: 628, Score: 1443.60, Max: 1491.93, Min: 45.66, T-max: 1999   \n",
      "Episode: 629, Score: 752.19, Max: 1491.93, Min: 45.66, T-max: 1000   \n",
      "Episode: 630, Score: 1467.17, Max: 1491.93, Min: 45.66, T-max: 1999   \n",
      "*** Episode 630 \t Average Score (over agents): 1467.17 \t Average Score on 100 Episode: 990.01, Time: 02:45:19***\n",
      "Episode: 631, Score: 1438.27, Max: 1491.93, Min: 45.66, T-max: 1999   \n",
      "Episode: 632, Score: 1484.94, Max: 1491.93, Min: 45.66, T-max: 1999   \n",
      "Episode: 633, Score: 1494.47, Max: 1494.47, Min: 45.66, T-max: 1999   \n",
      "Episode: 634, Score: 1509.01, Max: 1509.01, Min: 45.66, T-max: 1999   \n",
      "Episode: 635, Score: 1492.49, Max: 1509.01, Min: 45.66, T-max: 1999   \n",
      "Episode: 636, Score: 775.77, Max: 1509.01, Min: 45.66, T-max: 1000   \n",
      "Episode: 637, Score: 1536.44, Max: 1536.44, Min: 45.66, T-max: 1999   \n",
      "Episode: 638, Score: 744.08, Max: 1536.44, Min: 45.66, T-max: 1000   \n",
      "Episode: 639, Score: 767.02, Max: 1536.44, Min: 45.66, T-max: 1000   \n",
      "Episode: 640, Score: 1533.60, Max: 1536.44, Min: 45.66, T-max: 1999   \n",
      "*** Episode 640 \t Average Score (over agents): 1533.60 \t Average Score on 100 Episode: 1027.08, Time: 02:48:54***\n",
      "Episode: 641, Score: 1440.44, Max: 1536.44, Min: 45.66, T-max: 1999   \n",
      "Episode: 642, Score: 1464.89, Max: 1536.44, Min: 45.66, T-max: 1999   \n",
      "Episode: 643, Score: 1533.26, Max: 1536.44, Min: 45.66, T-max: 1999   \n",
      "Episode: 644, Score: 747.24, Max: 1536.44, Min: 45.66, T-max: 1000   \n",
      "Episode: 645, Score: 1507.20, Max: 1536.44, Min: 45.66, T-max: 1999   \n",
      "Episode: 646, Score: 1504.27, Max: 1536.44, Min: 45.66, T-max: 1999   \n",
      "Episode: 647, Score: 1582.90, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 648, Score: 1517.87, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 649, Score: 1452.17, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 650, Score: 1473.29, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "*** Episode 650 \t Average Score (over agents): 1473.29 \t Average Score on 100 Episode: 1076.17, Time: 02:52:52***\n",
      "Episode: 651, Score: 1506.89, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 652, Score: 1524.53, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 653, Score: 802.59, Max: 1582.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 654, Score: 798.10, Max: 1582.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 655, Score: 1566.07, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 656, Score: 1481.63, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 657, Score: 1547.38, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 658, Score: 1552.06, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 659, Score: 1487.45, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 660, Score: 1504.01, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "*** Episode 660 \t Average Score (over agents): 1504.01 \t Average Score on 100 Episode: 1117.92, Time: 02:56:35***\n",
      "Episode: 661, Score: 1546.02, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 662, Score: 1568.64, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 663, Score: 772.19, Max: 1582.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 664, Score: 1569.20, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 665, Score: 1555.07, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 666, Score: 1548.27, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 667, Score: 1411.70, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 668, Score: 1504.43, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 669, Score: 793.91, Max: 1582.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 670, Score: 1539.62, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "*** Episode 670 \t Average Score (over agents): 1539.62 \t Average Score on 100 Episode: 1170.05, Time: 03:00:21***\n",
      "Episode: 671, Score: 1513.28, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 672, Score: 795.60, Max: 1582.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 673, Score: 756.06, Max: 1582.90, Min: 45.66, T-max: 1000   \n",
      "Episode: 674, Score: 1543.69, Max: 1582.90, Min: 45.66, T-max: 1999   \n",
      "Episode: 675, Score: 1607.51, Max: 1607.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 676, Score: 797.16, Max: 1607.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 677, Score: 1526.42, Max: 1607.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 678, Score: 1514.77, Max: 1607.51, Min: 45.66, T-max: 1999   \n",
      "\n",
      " End of PPO!\n"
     ]
    }
   ],
   "source": [
    "def ppo(env, agent, n_episodes=2000, print_every=10, n_agent = 12):\n",
    "    \"\"\"\n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        print_every (int): number of episodes to print result\n",
    "        n_agent (int): number of identical agents in environment\n",
    "    \"\"\"\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores_global = []    \n",
    "    scores = []\n",
    "    max_t = 2000\n",
    "        \n",
    "    time_start = time.time()\n",
    "    \n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        states = env_info.vector_observations\n",
    "        agent_scores = np.zeros(n_agent)\n",
    "        t_max = 0\n",
    "        for t in range(max_t):\n",
    "            actions, log_probs, _, values = agent.act(states)\n",
    "            # get needed information from environment\n",
    "            env_info = env.step(actions)[brain_name]\n",
    "            next_states = env_info.vector_observations\n",
    "            rewards = env_info.rewards\n",
    "            dones = np.array([1 if t else 0 for t in env_info.local_done])\n",
    "            agent.save_step([states, values.detach(), actions, log_probs.detach(), rewards, 1 - dones])\n",
    "            states = next_states\n",
    "            agent_scores += rewards\n",
    "            t_max = t\n",
    "            if all(dones) or t == max_t-1:\n",
    "                agent.step(next_states)\n",
    "                break\n",
    "                \n",
    "        score = np.mean(agent_scores)\n",
    "        scores_deque.append(score)\n",
    "        scores.append(score)\n",
    "\n",
    "        print('Episode: {}, Score: {:.2f}, Max: {:.2f}, Min: {:.2f}, T-max: {}   '\\\n",
    "              .format(i_episode, score, np.max(scores), np.min(scores), t_max))\n",
    "        \n",
    "        if i_episode % print_every == 0:\n",
    "            torch.save(agent.actor_critic.actor.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.actor_critic.critic.state_dict(), 'checkpoint_critic.pth')\n",
    "            s = (int)(time.time() - time_start) \n",
    "            print('*** Episode {} \\t Average Score (over agents): {:.2f} \\t Average Score on 100 Episode: {:.2f}, Time: {:02}:{:02}:{:02}***'\\\n",
    "                  .format(i_episode, score, np.mean(scores_deque), s//3600, s%3600//60, s%60))\n",
    "            \n",
    "        if len(scores_deque) == 100 and np.mean(scores_deque) >= 1200: # 800: # 650: # 500:   \n",
    "            break\n",
    "\n",
    "\n",
    "    print('\\n End of PPO!')\n",
    "    return scores\n",
    "\n",
    "scores = ppo(env=env, agent=agent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEKCAYAAADq59mMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztvXmYHGW1+P85PUtmJvuekB0ICXsSwib7vgqIIqAicrmXnwruVy+oF9xQvPoFxQUFQeEqq3IFBcGwL7IlIWFJIAkkJJN1siczSWbp8/ujqrqrqqt6m+numcn5PM88U/XWW1Wnw/CePusrqophGIZhFEKi0gIYhmEYPQ9THoZhGEbBmPIwDMMwCsaUh2EYhlEwpjwMwzCMgjHlYRiGYRSMKQ/DMAyjYEx5GIZhGAVjysMwDMMomOpKC1AKhg0bphMnTqy0GIZhGD2KOXPmrFfV4fnM7ZXKY+LEicyePbvSYhiGYfQoROSDfOea28owDMMoGFMehmEYRsGY8jAMwzAKxpSHYRiGUTCmPAzDMIyCMeVhGIZhFEzJlIeI3CEi60TkrdD4F0TkXRF5W0T+xzd+jYgsca+d5hs/3R1bIiJXl0pewzAMI39KaXn8ATjdPyAiJwDnAgep6v7AT93x/YCLgP3de34tIlUiUgX8CjgD2A+42J1rGIZh+HjkjdVsam4t2/tKpjxU9TlgY2j4c8ANqrrLnbPOHT8XuFdVd6nqUmAJcJj7s0RV31fVVuBed65hGIbhsmJjC1fePZcv3zevbO8sd8xjH+AYEXlFRJ4VkUPd8THACt+8RncsbjwDEblCRGaLyOympqYSiG4YhtE92ehaHBuad5XtneVWHtXAYOAI4OvA/SIigETM1SzjmYOqt6rqTFWdOXx4Xq1ZDMMwuiWrt+xg0dptec9vbm0HoG9t+TpOlbu3VSPwoKoq8KqIJIFh7vg437yxwCr3OG7cMAyjV3Lkj54CYNkNZ+U1v3lXBwD9+pRvSS+35fFX4EQAEdkHqAXWAw8DF4lIHxGZBEwGXgVeAyaLyCQRqcUJqj9cZpkNwzAqSuOmFto6kgD8bf4qvnTv6wHLpHmXa3n0BuUhIvcALwFTRKRRRC4H7gD2dNN37wUuVYe3gfuBBcBjwJWq2qGq7cBVwOPAQuB+d65hGMZuwdadbRz946f577++RUdS+cI9r/PQvFWcetNzqTnbKqA8SvYmVb045tKnYuZfD1wfMf4o8GgXimYYhtEjaOtI0truWBz3vraCwyYNCVzvSCpVCUlZHv36VJVNNqswNwzD6KY072qnOpHOG/rq/fMD159+Z11qHkC/PjVlk82Uh2EYRgVRVf4yp5EtO9pobU/SkUwnlG7b2Y5G5pc6vLBkPYde/wTLNrQAUFMdlaBaGnrlToKGYRg9hZfe38DXHpgfeW37rnYaauNdUXe9tIykOkF0cIoFVRWnAqK0mOVhGIZRQbbvbI+99r8vf8AhP3gi9vqoAXWB83teXcHNTy7pMtmyYcrDMAyjm3L3K8uzXh9QnxnjuOmJRaUSJ4C5rQzDMCrEbc+9z/WPLiz6/nfWZFahjxzQpzMi5Y1ZHoZhGBWiM4ojju+ec0CXPzMKUx6GYRhl4vG313DBb/7F+03b2dHaUZJ3lCFWDpjbyjAMo8tRVZpbO1K9pm6ctYhB9TW80biZ15Zt4l/vbWBwQ21J3l2uZF2zPAzDMLqYO15cxgHXPc6aLTtRVW5+cjHf+/sCvBIOBVZsain4uXO+fTKDGrIXAibKZHqY5WEYhtHFPPKGU3fx+vJNgfEdbY6rauP21oKzon780QMZ2i93MNzcVoZhGD2QNxo3M3f5ZgA+96e5gWurNu8A4HcvvF/wcyVPh1S5lIe5rQzDMLqQ+2eviL3mtRoZlocFESZfpZCvkukspjwMwzC6kGwxh6SrPZaub875nN9ecghnHDAqdZ5vyxGzPAzDMHogXbV2n7b/KE7b36c88n1/mbSHKQ/DMIwuJNviHe6Q+9tLDsnxrOjjrPfkN63TlHInwTtEZJ27a2D42n+KiIrIMPdcRORmEVkiIm+IyAzf3EtFZLH7c2mp5DUMw+gKvL01olCC2mPc4Ia8n5u38ugFbqs/AKeHB0VkHHAK4O/4dQbOvuWTgSuAW9y5Q4DrgMOBw4DrRGRwCWU2DMPIix2tHazbtjNjfPOOtth7wpZHIscK7Ldi8q3fKFedR8mUh6o+B2yMuHQT8A0IqOBzgbvc/cxfBgaJyGjgNGCWqm5U1U3ALCIUkmEYRrm54n9nc9j1T6Lq37ypjdnLNlKViF7AkyHtkWuhj3kMAHsMrIsc7/FuqyhE5BxgpaqGdz4ZA/jz2xrdsbhxwzCMivL84vUA/Ou9DamxDza0sKmljV9ePD2wfew3z5wKQHhTwGzKAYJpt+FYytkH7xF3U1kom/IQkQbgW8C1UZcjxjTLeNTzrxCR2SIyu6mpqXhBDcMw8qC22lk+r7o7XQjoGRY1Veml9dtn7cuew/oFrqcJLnFhZRIImIfvjFESvbHOYy9gEjBfRJYBY4G5IjIKx6IY55s7FliVZTwDVb1VVWeq6szhw4eXQHzDMIw0g90eU35F4QXEg1lSEr/Qh8bDbizJMrcq5qG5rJmuomzKQ1XfVNURqjpRVSfiKIYZqroGeBj4tJt1dQSwRVVXA48Dp4rIYDdQfqo7ZhiGUXI+2NBMY0QDw/aOJE3bdgEEGhV6lkXYYvDONUfMI0N5+M7DFkVcXKVcdR4l620lIvcAxwPDRKQRuE5Vb4+Z/ihwJrAEaAEuA1DVjSLyfeA1d973VDUqCG8YhtHlHPeTZwBYdsNZAHz4Fy/Q3NrOzRdNT3fI9ekDLyAezJJKnydDbqvwMp9IAL5tPvx6INOlFac8Yj9Ol1Iy5aGqF+e4PtF3rMCVMfPuAO7oUuEMwzBy4E/DXb1lB6MH1vPmyi0AzHW75R48blCgrsPTDUF3U9pmCGdbdcZtFeee6pXZVoZhGD2F5RvS7qotodqNpm27EIGxg+oDrqi028qfJRVvJeRyWwXPs89Nv6+Hu60MwzB6Au0dST77xzks39jCXsP7cfnRkxgzuD7gYgpnSe1o7aC+pgqR4DVPkfitAr/lkZltFSS87mdrTxIf88j+jq7ClIdhGLs1Kzbt4ImF6wBYtHY7/3hrDfU1VfzhskNTc8Luph1tHTTUViEigdqBtNvKH+hOWwnhgHkuhVBcqm55MLeVYRi7Fb999j1OvenZrHN2tHUElULY8mjroK6mioQQ47ZKz3XcVs5xRsA8pAHC6bd+JRR2U8Wl6prbyjAMowT86B/vBM7D1oBH2Nrws7PNdVsRVAipbCvf3ITfbUU4VTf43PDCX4zbqtfVeRiGYXRH5jdujr4QoRQ8drR2UJ9yW+UImPvOO5LBV4RrN7Ipk8x4SIzl0QsrzA3DMCrG1p1t/O/LH6TOr39kAdt3tfOV+8Kt9hyyua1aWh23VUbAPLLCPL5IsKBU3RyKJu6ZpcLcVoZh7BZc85c3eeTN1anz255fmjX7yW9thKftbOtgUEMtggSf4VkeviF/tlVHWHmEnpstYB6eXK7W63GY5WEYxm5BVJuR5taOiJkOUZXjHjva/Km66pvn/E4kot1WyWTY8ghZE6EV2a8gMmpAKpyqa8rDMIzdgihF0dqejJjpkCvbqqHWCZgHDY/MgLmIpFxMmdlWwfPsbqvw3Gi5y2WRmNvKMIxexeK129jU0sZhk4YAsGrzDtZt20VLxPawreEIto9klD/KZUdrkrraKjo6NFQk6PwO96RKp+pmd1tlLPzZsq16a28rwzCMSnDKTc8BTjPDe19dztUPvgkEu996tLbHu62C2VbBS7vaO6itSrAz2RFQCOnjcJaUhK47ZHbRDb4nWGyYZ3sSy7YyDMPoHHe+lM6u2tySubd4drdVZgqu/zzh7tMRZZ8E2pPgc1uFU3VzuK0SWSwPi3kYhmHkwa+eXsLC1Vvznr+jtYPp4wdlnbMri/LwL/ThFFtVTbVaj8y28rUt8TdGzMy2ylFhHqoX8RMf84ge72pMeRiG0e1RVX7y+LucdfPzed+zfvsudrZlcUuRXXn4l/mw2yqprlIgnG0VHTCX0PX0xeBp2JoI70gYmBtrYpjbyjCM3ZxkUlHV1OIdXsTBqbn4zbPvsSsUv9jQ3MqOLKm43r1xBHpWhQLmika7rbxUXZ/CcFJ1g9c9wlZCRoW5/3h3cVuJyB0isk5E3vKN/URE3hGRN0Tk/0RkkO/aNSKyRETeFZHTfOOnu2NLROTqUslrGEb348ybn2fmD56I7T8FcNOsRdzwj3d41FcACLCppZUdOSyPbMojGRXM8F8TR0loRDGhX6kkRPLeeyP7NrSE5kbLXa7SwVJaHn8ATg+NzQIOUNWDgEXANQAish9wEbC/e8+vRaRKRKqAXwFnAPsBF7tzDcPYDXhnzTY2NLdGWhwery1zdqZu78iMS7TksDyyper6NUbG+zVtXQT3/cgUNJslkEshZHNbxaXqlqvOo2TKQ1WfAzaGxv6pql6y9cvAWPf4XOBeVd2lqktx9jI/zP1Zoqrvq2orcK871zCM3Qi/2+i8X73IsvXNqXOvpceG5tbAPclkdssCYFdblphHRM+q1LNVU5XjkRXmMW6rMLmaHWZzW1V6D/NKxjz+DfiHezwGWOG71uiOxY0bhtFLWL99Fw/NW5kx7l/4/Qv5vBWbeWdNOuuqrqYKgNueez/DhZTL8siabZVlJ0ElOlXXO/OPOwHz/KyEzN5W/vYkZJ2buqc3B8xF5FtAO/AnbyhimmYZj3rmFSIyW0RmNzU1dY2ghmGUBFXl9y8uZevONv7jrtl86d55rNu2MzBn3dZdvvnB+59dtJ63Vm4JjG1obmX5xnT/qqRq5wLmAbdVhOUhZDRGjNsMKtyzKo7sAfT4GpDArN5qeYjIpcDZwCc1/TWhERjnmzYWWJVlPANVvVVVZ6rqzOHDh3e94IZhdBnzG7fw3b8t4OsPzGfNFkdptHcol9z+Chff+jIAbb5Ci/Difc+ryzn7Fy8AQcXS7jMXVMkZMG/LEvMIuq0yr4lneUS4rcLb0MZZA7l7W4Ur1ePnxs0rFWVtTyIipwP/BRynqv4Wlw8Dd4vIjcAewGTgVZx/98kiMglYiRNU/0Q5ZTYMo+upq3G+t77X1BxYpJ9fvD51HJXFFEWgdXooeJ3L8sj3uVETxf2Jaozotwo891YUmRXlWeo8wvfGpur2cLeViNwDvARMEZFGEbkc+CXQH5glIvNE5DcAqvo2cD+wAHgMuFJVO9zg+lXA48BC4H53rmEYPYi3Vm4JuIhqqpylZ2MoyO0nW0v0uHlhhROu6M52bzai3p8QIZHI022VZ1ptNveWpxTStST5PbNUlMzyUNWLI4ZvzzL/euD6iPFHgUe7UDTDMMrAw/NXUV9TxYFjBnL2L17ggkPG8pMLDgbSi/zG5lZGD6zLuHdXewertqRjIMVs2pRUzVofkosopeDHqzAvpDFi5jOyWx7+8/ATrKuuYRi9ki/e8zoA3z5rX8DJkvLItaZ/4e7X+eeCtb75WSwP33HYhdUJ3ZF1J0Fwv/lnZFv5rrlIFrdVZp1HvNvKu5Yay9MVVipMeRiGUVJ+8MjC1PHL729gcENtznv8igNyKAG/e8sX/06qZnV35SKX20y8yvFIt1U4YB5NZhA8/nq2du3B8fJgysMwjLJxkZtJ9Y8vHZNx7SO/fjH2vmxKwH8tnD2VrTI9F1E9q/xEua3idxLMz22VUeeRRRVUuC+iNUY0DKPzzF2+iVfe35D3/KjFeK2vriNjfrZn+Y797UY6Y3Vk3h9heZBZJOhZPuEsqXw9SZkxEP+18PtjnlEm7WGWh2EYneb8X/8LcHbvi2Pxuu2pY//C3J6HeZCv5eHf3Clr26p8CLitMi8nsjRG9FsaiSwV5lHP9BNUQvHV59meUSrM8jAMI2+2tLSx1zcf5el318XOadq2K6P6Oxvbd2buLZ6xLmbRL3694lceyc74rMi+kyAU1hgxX8sjs+WITwklgrLEV5ib5WEYRjdjSdM2OpLKTbMWccKUERnXO5LKUT9+Kuv2rhC0FnJVgTvz46/5F2x/ryqvxqMqIXQUoUgCSiHKbRXhj4qs88icFktWt1WG5RHzjPxe1WnM8jAMI2+8BTWu4eB3Hn47p+KAwlNooxbv9LX0Qtqe9LutXOVR5DdxzeG28u8Q6CmwdIW5BOflKUPW/Twk+Du+diSvV3UaUx6GYeTN1h1tAIG2H//vn++mjv88pzGv5xQazM5mOCRVUwrCb2F478i3KWHUcz0i3VGkF3vvclyFeb7redadBMPvr7DbypSHYRh5s3Wnozz8rUZ+8dSSgp/zETfAni9ZiwQ13efJv+B32vLIcT3hi2V4741rjJhv4V5Y1ux1HtGY5WEYRrfh7leWM/HqR1i81smY2tDcSvOuzEB3V5Gxf0ZWywOf5ZEe95RHXAPBQoSIKxJMua1SvyMaIybiK8zDhGUNKp38sq0s5mEYRrfgmXfX8c3/exOA15enW4y87NZ1nL7/KABGDuhTMhmyKQ9VTWUpdUS0cY/bNCkX2TaDAjdVNxHttvKv4NkqzKOeme+1eMvD3FaGYZSRNxo3c9dLyzLGn34nnZbrua0gXbdRU+0sI4Pqa0vmMsnVVddbWP01I95xdZHKI2qfjgC+D5tMBczdS+HGiPlaHqH+VVF7mEfFVYLPyO9dncVSdQ3DAOCcXzrtQT595MTA+E7fPt+rt+xkwtAGWlo7Ui4sTfn7O1dXkY3sFeZpy8Nf2+EdF9soMNieJKole+YC7s0LL/r5ypCxh3mWrrqxLU968za0hmH0HHa2p4PjG5tbmTi0L2MG1ae2jU25bEooQy7Lw1MefsvDi38UqzyCdR6ZCBKbbRVupV6o2yqqEDDKGonCAuaGYXQLwrvxTRrWl+qE+DKMymB55EjVTaQC5j7l0cmYR6DtSJzl4ZPBPy8Q5i6gziNbY8R8iwTLRSl3ErxDRNaJyFu+sSEiMktEFru/B7vjIiI3i8gSEXlDRGb47rnUnb/Y3f/cMIwysjNU9Dd2cD0i6SaAqXW1hKZHrv08qiJSdT23VbHKI/j+zDF/2xHvcipV159tVVCdRyGputFPLdd+HqW0PP4AnB4auxp4UlUnA0+65wBn4OxbPhm4ArgFHGUDXAccDhwGXOcpHMMwysPOUPuQkQPqEJFUWmpZLI9s1xSf5ZEe76zlkWszKPE1PNRwwFyCFkO+C3pGwDzwvvD7o5/R491WqvocsDE0fC5wp3t8J3Ceb/wudXgZGCQio4HTgFmqulFVNwGzyFRIhmGUkF0RyiMh6W/Z/t+lWreyxzxiUnVTAfPi3plzMygyLY/ogDkFZFuF3hFqcxK8Fv2Mcnmzyp1tNVJVVwOo6moR8TqrjQFW+OY1umNx44ZhdBFL1zdTVxP/PTLcuHDkgD4kRHwLdbCvUynIWSSYiLA8Oum2CmZbZV5P+GIZGnLhhS2GYosEww0WA9fy3Be9VHSXVN2oT6tZxjMfIHIFjsuL8ePHd51khtHLOeGnz1BfUxV73Z+qCzCsXx8n5hG2PDq7f0YWshYJolS5Daz8lofntio+2yqX2yq9QGlIgXbWbRWnhIJz456R16s6Tbmzrda67ijc3171USMwzjdvLLAqy3gGqnqrqs5U1ZnDhw/vcsENozcx8epHuGnWIpq2Obv3ZWuLvqs9eK2mKkFCMrOtSknWzaCS6fYkwVTdzmZb+Y+jsq0kI7U2LsU2XwmqQityWAkFr0U/o7dWmD8MeBlTlwIP+cY/7WZdHQFscd1bjwOnishgN1B+qjtmGEaReAHwnz+5mMXrtgHQpzq9FIQXynCLdac4TjIWzHIokTg8d0+Hdp3y8BOfbRXM8vJXmKvP8snbbZVlP4/Mj1HZXN2Sua1E5B7geGCYiDTiZE3dANwvIpcDy4EL3OmPAmcCS4AW4DIAVd0oIt8HXnPnfU9Vw0F4wzAKYP12x9roX1edWmBrqxKpjZRUs/vovW/c4cryUiqPXNvQet/Yk1Et2Yt1W/meFb1DoGQEzJMxAfN8q749ZZTOtooPelS6zqNkykNVL465dFLEXAWujHnOHcAdXSiaYezWrN/eCsDQvrWRPZvakkn6JOJjICK4bqvguKN0SrOiZW+MGJOq24UB86RmKhAn2yq6wjxllagWtJNgll1oM91W+T2yZFiFuWHsZqx34xxD+tZGfqOe8u3HeCbLHuVeK/LMCvOul9Ujl+WRVh7+nQSd312xk6CSqcC8PcyduWHFknZbSSfcVlmLBCtsepjyMIzdAP/itqHZrzyi5z/zblP63ojr/phHer2ujNvKX2Huj3l0viV70G0VliCywjzZObdVWNZwj6zA+/N6Yukw5WEYvZAl67anrIdbn3uPw374ZKpHlRcA71NdlV4gC1yJnCLBYHpqKS2PjixpwKpEbkPb1XUeYQWW8CmFcHNIfw1IIW6rbLsFZgbTe2nA3DCMynHyjc8C8P8uOJgfPvoOAK+v2MSH9hoWWOSLXfATfsujDNlWHVkEVdXUPuVRjRGLXmMDdR6aYaWJP1XXq/Pw6WJ/tlVJtqGtsOlhlodh9GK+9sD81PFjb60BQs0Di1zwxW95aHDhLAW53FbVrvbw13l4LqRiN4MK7yQYlsFvUYQVaKbbKj8yYh5ZuupWmryVh4gcLSKXucfDRWRS6cQyDCNflq5vzmibHsVfX18J+BZFyQz0RhHXmiMZUhpJN7OoFGSzPJKqqTqPZJe6rfxKNvN6IkdjxLTbKv+AeVbroidaHiJyHfBfwDXuUA3wx1IJZRhGfqgqJ/z0Ga7439k553rfynNur5oHIpm1DaVsyd6RzfJQqBJvXnq8s3UewWyrKLeVL2Cu6ZvSY+qbV1h7Ev87oo6d856RbfUR4BygGUBVVwH9SyWUYRjZuefV5dz63HupRev5xetz3hNVzFes26rcMY9kDssjqqtueyctj1xuq0BjRJ/+zMiKkuDvbGTfDCr03NyPKyn5BsxbVVVFRAFEpG8JZTIMIwu72ju45sE3Abj86D3zvi/cyDB8XAjBbKv0s0rVlC+b24pAkWCm2yrcqTZfFE21no9K1cUXy/DvbRK3D7mQ2zgLixpsTxJvlVSCfC2P+0Xktzj7bPwH8ARwW+nEMgwjjlWbd6aOC/m2H24n4h/L5z4/Eoh5BFN2S0Hu9iSZyiNV51HsKutTSqqZ/w4JkVSWl7/CPE4B5ONmymzJ7rM8Mp7bA1J1VfWnInIKsBWYAlyrqrNKKplhGJH4F9L3mrYXcJ/zO9gttjgZ3M4bAXkqVueBrzFiFwbMU4H4pKMWw59PSLuV/FZYZhsR5zwhkCutITPbKvM5UdcqQU7lISJVwOOqejLOTn6GYVQQ/zfg03/2fN73hS0Ff4uRQvHHPNLB4qIelRfZAuZJ1XSRoN8l5yqczgTMvecmVTM+n79bbiB5ICYrSvJwXGXuJBh97D6wouR0W6lqB9AiIgPLII9hGDko9hu+53qJinkUug75Yx6VDpj7XUWBVN1Ue5Li3qmkn+vojlCdhy+Lyq9AY9f4PP6RM+Ma8TdVuu4j34D5TuBNEZmFm3EFoKpfLIlUhrGbcOOsRdz10jJe/+9T8k697MwiHc4aCge9I++JGBMyYx6VqzAntZNge6AxYhe4rTzlQITbyh8w9/07ximAfKRIZJnb3QLm+SqPR9wfwzC6kJufXAzA3OWbOGTCkNh5zy9u4kN7DaMqIZ3a7jUZsjzyCZhHkUhkxlCcVMziZctG1joP0vt5RAXMO1XnIV58RyM750Y1Rowr9MtHjkQWKylb36tKkJdBp6p3AvcAc9yfu90xwzA6wdRRTrnUuq27Yuc8u6iJS25/lVueWQIU9w3fa9Hh35fCsUKi5+dSKoGuuuVoT5LT8sgMmLe7AZBi25NAuu16RMgjtjGi97awxZGPDvNbOmEy60d6gNtKRI4H7gSW4XyGcSJyqao+VzrRDGP3IVscY+1WJzV32YYWoLhFuiohtCfVtTzSbqa8FFHEFP9OgppjbleQK2AeVeeRsjw64bbyNr5SMv+t4hojptN7g5ZPIW6rKDLqR/L5ECUk31DS/wNOVdXjVPVY4DTgpmJfKiJfEZG3ReQtEblHROpEZJKIvCIii0XkPhGpdef2cc+XuNcnFvtew+iu5LOIe1OyLaRxeN/M/dZGNssjF/7eVl1RsZ6LrJYH2ffz6IzbSvCaQBKRbeVrjJj0vbMzbqusAfLuRb7Ko0ZV3/VOVHURTn+rghGRMcAXgZmqegBQBVwE/Bi4SVUnA5uAy91bLgc2qereOArrx8W81zC6M0lVJl79CN/+65sZ18KLRjELdFXKbRW0PDyNVOjC5KT5OsfBHlClIVfAPGobWq89SbFuK6fCXNxdATM/m/NKSc1NjYfneSN5iOHFbqKmVjpAHiZf5TFbRG4XkePdn9twYh/FUg3Ui0g10ACsBk4E/uxevxM4zz0+1z3HvX6SVNrZZxhF8PsXl/LBhubIa/e9tgKAP768POdziglyV/uUh78nVdyanOt/MSfmEcy28p5fCjpiHuu9O6UcI1qyF++2Sjc/zOm2SilSzawSz6IQwmRNze1my16+yuNzwNs4FsOXgAXAZ4t5oaquBH4KLMdRGltwFNFmVW13pzUCY9zjMcAK9952d/7Q8HNF5AoRmS0is5uamsKXDaOiNO9q57t/W8CFv305NTbng41sbG4F4F/vbcj7WcW4mrxU1qT6F9j8Yh5RM6IaI0LpguZxbitv2FMegVTdTrYnUTd9zKumD382x6UVHzD3z4P8Fv9KtxwphHxTdauBn6vqjZCqOu9TzAtFZDCONTEJ2Aw8AJwRMdX7TxX1r5nxl6SqtwK3AsycObOEeR+GUTjeH+S2nW2psY/e8lJRz8rm/4+jOhXzSKfq+i2P8BNzWTfBIsH03FIpj7g4TzgoHSiA9CrMi8620lRGlUYkF3iZWM7M9L9FXGPEfMRIB+DTvPqtk3hn9bbI+SdOHcFT7zjbDU8c2sC1H94v90u6iHwtjyeBet95PU5zxGKtaUM1AAAgAElEQVQ4GViqqk2q2gY8CHwIp+mip8zGAqvc40ZgHIB7fSCwsch3G0aPpzjLI724+ov7im+MGBfzKJHbKqfl4fz2LI+E+IoEi/w2n0ziZlt5Kc7B6yJENkYMv60zjREBRvSv49h9hkfOv+Mzh/L7yw4FYGi/Ppw4dWTOd3QV+SqPOlVNdWBzjxuKfOdy4AgRaXBjFyfhuMGeBj7mzrkUeMg9ftg9x73+lBZb2WQYFSL8J9uZP+GiYh5V/piH9y3Zt21qgc8L7iRYessjztrylFWqt1WqtiPRBe1J1C0ElMjP5d9JMNAYMWYr2bh/4z2Hp3e48D5Htv8e3pa7nkjD+jpOoA3b42uFSkG+/6zNIjLDOxGRmcCOYl6oqq/gBL7nAm+6MtyKs1PhV0VkCU5M43b3ltuBoe74V4Gri3mvYVSSsHuomEU27Rop/F5/tpV3e7jaPP69mYhI+rP4xksXMI9zWzm/E6FU3UQiej+PQowQr2eWEB0wdx7ozk3dE19hHmd5/O2qo1PH+bRSuec/DueqE/ZmcIOT8Dqsfy0AG7a35ry3K8k35vFl4AERWYXz77QHcGGxL1XV64DrQsPvA4dFzN0JXFDsuwyjOxC2Fgqp1QgvOkWl6voCu12zk2B0T6tCFVtVQrJv9OQ9N87y8NxWoVTdKpFIt5WjCPLDybaKD5gHYh5Z3Fa5dhLs2ye9DI8bnNuhM3lkf/7ztCmp86Gu5bFtV3vcLSUhq+UhIoeKyChVfQ2YCtwHtAOPAUvLIJ9h9ArC3Ws78w2983Ue6efkVWAe47KJ2pmwYLnybgYZNx60LvznnsLxf5svJJvJs9G8tOSwCImE/3meCy9qTw7n3PusE4bGK4jRg+ryls+jtjrB54/fiwc+e2TB93aGXJbHb3EC3ABHAt8EvgBMw3E1fSzmPsMwfIQX/M40NyxG76RjHj6LIZlf/CQqCC6+bKvOeKoSCXLvkEQWt5X7O5Wq65oeVQmhtT0ZuAYFFtqpI59T55H539DfGNGvkOPcVoMaalizdSfnHrwHlx8T3D54YH0NW3a0UZ0QZowfxBXH5r+9MMA3Tp9a0PyuIJfyqFJVL7PpQuBWVf0L8BcRmVda0Qyj95ChPIoLehR9b6rOI6kpxeX48Z3jsGssGMfIfJ4/iNyZ4L8T/M2tSePdVkHXlDetOiHsiLA8nM+Zn7xJdQLmXnJAptsqe2PE1Dvd3wPra1IyeMcez339BNqSSUSEBz9/VF7yVZpcAfMqX/rsScBTvmv5xksMo1fz1sotrNu2M2O8rSPJ9Y8sYFNza8bCU0x/Ko9i3ETpOo/gJk5RGVNhoq6lK6vzbK4YQ74lGLlSdT3dl07VTWeDJUIxj3xR0nt2OI8KWR4S/HfAnecp4kMmDHbnOedtrlU0ckCma2pgQw3D+hVVOlcxcimAe4BnRWQ9TnbV8wAisjdOpbdh7Pac/YsXGFBXzRvfOS0w/s+313Lb80vZsL2Vr58+JXBNi3FbpVwkXRPz8BcMZnti1LrtL8pTHOXUXoRWy3ejplhl62VbuYFtz6qq9gXii3VbefELL7MsczMoSf03eWBOI9VVTvdd7x13fOZQPtjQknp/8y7HPzd+SLFVDt2LrMpDVa8XkSeB0cA/ffUVCZzYh2EYwNadmZku3rfgtmRmSmxRlkcn3ETVvlRW/7dk/3EcUcrKH/hPJt1+TkUoj3wD2PHtSbygtvOslOWRSAf0/fqpkIC547bybwYVvC6k3VZ/ntPIn+c0MnFoQ0p59K+r4YAx6d27d7U7ymOPIoLi3ZGcridVfTlibFFpxDGM3kl48SvEeggvd52p8/C7mTI67EYQtWhCOrvJS2Ettoo732Z/ccrW75oS0g0U/dZGlW97vkLdVvjcVuHEAW+fDz/LNrQwY/ygyOfdfPF0/vDiMiYM7Rt5vadhcQvDKAPhta+Y/lSpezvltvJnBkW3GAEy6hfCpLOMNLCfRqHkH/OIGQ9ZHp555ldm/grzgiyPpNOSPSHCva+t4B9vrQlcF4GWXZmpYhcdOj7yeQeNHcSNF07L+/3dnSIL9w3DKITMbKvOPKvwe6oDMQ/XVeWTKyyfdxanqNK75Xk7+RUuE+Qfg4h1WyW950jArIit7ShAzo6k47ZqdTXXlh1tgesJEU6YOoLzZ4wJjI8ZXM/ugCkPw+hCPtjQzIqNLRnjXZGq691RTMwjvd9FcO8JYiwP9VknUST8lodCdZENpPK1BOLcVinLg6AVE3RbFZdt5dVseG3zgYwU26qEcOPHp7HshrNSY0P61hbwlp6LKQ/D6EKO+8kzHPM/T2eMZwTMy+y2qk7t55FfzMPfYjwKf0NA/x7ihdLpgLnXv8rXpNDZwMmnPPypugXI+cTCdYzoX0ef6vQyWVeT4NCJTgpuW4wvbehuojws5mEYZSCzq27+93ZFdXpVRJ2HBuIf0W61XDEPdecU27k237U81vJI+mMe7jMJWhjF7+cBJ+87goWrt7Kr3bE+powawM8unMaDcxuZNCw68D2oYfdQHmZ5GEYZ6EyqbkawvROpupnb0KbjH8F3Zrc8UjGPpGOlFJ9tld+8OEst1UXXZ20kRPAlWAVkq6vJXPJO2z96D4xPHD6eTx4xgcuOmgjA9887gJsvmsaQvrX8+zF7ZlgxP73gYA6bOITa6t1jWTXLwzA6Qb5ZU52JeWS4lDqRqhsuEtQYC8OzbuKVR/p6UqGqqsRuq7hUXc/y8PWZEl/bEAjGPAbW17B2a3Dfi9ED66mtTtDanuTsg0bz9zdW89zXT2C828DwyhP25t+OnkRDbfbl8mOHjOVjh4zN6/P0BnYPFWkYJSIfJSAR8wpJ1Q1PLcryiGiM6HdhhelIWR7Rz/O+dXsxj2Itj7wD5jnakyBpBdjWEWxO6HdbtbmFIMftM5zPHb8XAA21Vakmit86a19e+eZJKcUBzmfNpTh2R0x5GIbLY2+tYd6KzXnNXb6hhdb2ZFb3k/9Spuspf7m6Is03nVobHTCPe2dcZlciFPMoNq6Qr85p74iW44Ul6115hO2+/SzafPOrfbLtbHPqMq48Ye+UbdJQW5W6PrihNrL3lJFJRZSHiAwSkT+LyDsislBEjhSRISIyS0QWu78Hu3NFRG4WkSUi8oZ/R0PD6Eo++8c5nPerF3PO29TcyrE/eZrrHn4r7+B1eJEuJNtKQwt552IewSB5nBi5UnVTloc7obpY5ZHnvLiNjr7/9wWRz1m4emvqeFBDOr3WUx57DKqjpdU5rvdZFXU1aUViZKdSlsfPgcdUdSpwMLAQZ3vZJ1V1MvAk6e1mzwAmuz9XALeUX1yjN/DC4vVceffcTrUQB5j+/VkAPL94fdaF3P+tujOup/C9xdV5ZKbqapZn5Wpb4lky7b5U2WLI976toQK9MDvbozcF+ePlhwfagfzqkzM4YcpwRg+sZ4erPPrWVjGif8/qaNsdKLvyEJEBwLG4e5SraquqbgbOBe50p90JnOcenwvcpQ4vA4NEZHSZxTZ6AZ+6/RUeeWN1ZMB5S0v2xcnjR48uDJzn67bqyoB5URXmVekYhT9IHidGXAqvh2doRHWuLYTF67YHzm++eHrkvPXbd0WOezRt28VMtwX6N3wdjI+ePIy+PrfUh/Yaxu8vO4yqhPDJI8ZTW5XghKkjePzLx/Lc108o6jPsrlQiCrQn0AT8XkQOBuYAXwJGqupqAFVdLSIj3PljgBW++xvdsdXlE9noibS0tjPzB0/wi4unc9K+6XTMDlUSIUfHwd/7Z17P/O1z7wfO822trr5v+1CYAvAW6PS9xVgemXUe+cU8op8nIeVRjNvq5xdN40v3BveUq4tJc12/vTVy3GPdtl088NkjncwvdwvaZteyiCsMPGjsIBZdf0bqfPBuUtzXVVTCbVUNzABuUdXpQDNpF1UUUf/lM/6kReQKEZktIrObmpq6RlKjR7NsfQstrR385PF3A+Odqe4Ok2+9RrgYuRAF0Jlgu0dkbyuNf1au+Iq3IKfqLIpQHudOG5Mxlo8ba/KIfhn7gO83egAiklKSV504mf+qwNasuxOVUB6NQKOqvuKe/xlHmaz13FHu73W++eN8948FVoUfqqq3qupMVZ05fPjwkglv9Hz+uWAt//d6I+AEv+PaTORDNiXgtfjw7/eduq+gVF2vlsF9bmd6W2UEzLNXbsf3tgoGzAtN1e3fJ9rpUZ2lXqSfe099bRUnTU1bknf/++F88vDoTrZG6Si720pV14jIChGZoqrv4mxvu8D9uRS4wf39kHvLw8BVInIvcDiwxXNvGUYxfPGe1wE4b9oYpn9/FudO26PoZ2VTAqm9wjVTARSWqhs+70y2VXgzqOzvjPt8nqGRCpgXaHkMHxAdoB41MD5N1p+K63/dfnsMyNmz6jMfmkhzTMaWURyVqnz5AvAnEakF3gcuw7GC7heRy4HlwAXu3EeBM4ElQIs71zA6za1u/OKheRmGbN5kUwJ+l1Z4kS7EdRZuIVJUnUdgMyh8x9EPy7XDoLdWe4HsOMvj++cdwLtrtvLmyq3MX7GZfUcPYOHqran9uof1qw3EM8YMyq+d+VdO2YfbX1yKqrNjXy6+c87+eT3XyJ+KKA9VnQfMjLh0UsRcBa4suVBGr2XFxpZIV8+P/vFOp56rmj3mkWqdEeG2KsT1FJ7bKcsj6Q+Yxyui3NlWzvMuuf1VAFraolNlj5s8nEuOmEBHUulIKkvXN/Nvf3iN89x4x2vfOpntu9o58DtOwkI2RfDZ4/biN8++B0DfPtUs/N7prNmys+hML6NzWM290etpbu3gvtdW5J5YBNncVn7F0pnGiJl1HnnfmiJY54HvuLg6j7ZQxfdx+wxnfkR1/siBfdz3O8HsKaP68+LVJ6auiwj962o4ed+Rqb5QJ04dwVPvOCHPg8cNYv89BtCvTzVnHDAqpTzAKeibGNPZ1ig9pjyM3YI5H2wqyXOzWQH+Rb8ztRrhtNlissX8Feb43GDxRYLZ5VyzZUfgfFB9DR+dMZa/zG3k/BljeHDuSgD6VOdXsf27S9OOiDs+cyhzPtjE+CENDPcV70UpJ6NyWG8rY7fggTmNJXlutnXcb5VkuJ6KaIzYNXUe0V11wzy3qIn9rn2MJW4RX59Q/cWG5mDdhQgM7VebmvuFE/fmwwcXn4hwyITBAcUBpNJzLzliQtHPNboOUx5Gr6WYRbZQslkB/mvhHlgFVZgngy6kztV5hGMe8Q9rae3gs3+cA8CvPhFsKff54/fmwDEDU+fHTxnBYHcTpJ1tSb526hR+EVMtXiyDGmpZdsNZXDBzXO7JRskx5WH0eJZvaOFb//cm7aF6jTLojoBFkS2w3TVuq+DvQojbzyNfOfYZ2T9wPrx/H375ibRymDSsb8ryyNVKxOgdmPIwuj3H/s/T3Pzk4tjrX77vdf70ynLmNwZ94oUEpfMhatHOFhRPZrlWWKpu8J7O7eeRrvPw7+3h5/wZYxg3JJ0y269PNWMHZ6bQDqoPtvOYMMRxKy3f2FKwfEbPw5SH0e1ZvrGFG2ctir0et5R2ZRuSuOf53VHtId+UZwgJUXuYF56qmyuInY30fh4+t1VSeWLhusC8sw4czY0fn8bz3ziRQ9xGg6fsNzKyCLB/XXXg957D+wHwwQZTHrsDlm1l9GhWbGzh9eWOxZHZA6qLlUfE8wLWRUxcQ+lsqm7QXZXrc0Uppmo3Vfe7f3ubTW4H4ag9Mlb5sqiatjnuJ2+P79986hD2HpFOjU0khJ9fNI1p4wYBTsEfwDGTh+XxqYyejikPo0dz2/Pvx17rassjauMn/0IeVgj+jKrOxTzc53vddXPcG3Xdi3lsimg9f8KU4XywoYX31zezanOm8vDiHacfMCrjXn9zQxHh1W+elFfFt9HzMbeV0aPxL5ThNbOQdNh8CLulIKiggtlVmlImnd/DXN33K0vWbc95b5Rlkq1l+g/PP5BZXz2OhtoqvnZqei+M82c4imH8kIa4WzMYMaCO+lrbjW93wCwPo9fQFa3Ls5HL8vAv6h3+rKZI2QqJeTi/n1+8npNvfJZT9xsZOe+xt9YAGtnWvCpLt9rh/fpQlRAWfO/0wPh3z9mfq8+YSnWVfcc0MjHlYXRrcn3L1thweddnW0XHPKKvdyQ1JXtHMrMNSL4uteZd7Sxd3xwYW7ctOhXWq8mIot63N/dfPnckP37sXV5duhEgVjlUVyXob4rDiMH+MoxuTXsu5ZFnhXdXELXgh11VqWNNu638tRUe+eq1/3xgPs8uCm5uNs9t0/G54/eiNsvi/vGZY1NB7EEN6TjE0L592NziVIj//KJp+QliGCFMeRjdmkKC3uEsoy4PmOfItsqwPFIpsZn35msVvbBkfey1/zp9akAphFm1eSeT3MaBzbvSXW/711WzsdkJnM+cOCQvOQwjjCkPo1uTa5H1Xw3P7XK3VY46j4AVomlLpMNXmJe+Hi/bo2+uJplUHpq3km07C9/A6AfnHZCS5+aLp/OJw8dz0Nh0K5H+dTW0tjvKZKjt220UicU8jG5NR0f+bqvw4l4Ot1VcnUcyqakiQQ0Fz5374t/z+T/NjRw/aOxA3mjcAsCvPxnsNTW4oYZNLW3Mv+5UBtRVo6qctO9IRg+s54cfOTAwt7Y6wd3/cQRPvbOOuhrLjDKKo2LKQ0SqgNnASlU9W0QmAfcCQ4C5wCWq2ioifYC7gEOADcCFqrqsQmIbZSYqPTZIehUOx0e6PNsqR3sSv6wdvr0yogLmUYqtobaKltboTZUAHr7qaGYv28j4IQ2MGOBs1+olVv3hssNoqK1iYL3jxrrkyIlZP8sBYwZygK+xoWEUSiXdVl8CFvrOfwzcpKqTgU3A5e745cAmVd0buMmdZ+wm5HRb+S2PjtK6raKC9353VDhtN6U8NFNZtLZnKsXw2KVHTuD0/YOFeTMnDkkpDj8jB9QxOdS8MExUfyrDKJaKKA8RGQucBfzOPRfgRODP7pQ7gfPc43Pdc9zrJ0mu3e6NXkMhQe9sFd5dQdTzOpIxx6q+ivC028r7w93UEtwPA+CSIycEzr977gH85pJDqK2O/9/0v06fCsDgvrmruv/5lWOZd+0pOecZRj5Uym31M+AbgPdVaSiwWVW96GAj4PU9GAOsAFDVdhHZ4s6PT0Mxeg25lEe2mEeXN0bMlW0VqjaPc1u9sHg9v34mvZ2qx7fP2o+vnTqFl97bwMbmdC3Hy9ecxPaYwPn5M8Zy/oyxecnfUFtNg8XHjS6i7JaHiJwNrFNVf0VTlCWheVzzP/cKEZktIrObmpoibjF6IjmVR5aYR3myreLcVukAur/Oo7m1g0/d/kpq3nnT0rvtVSWEfn2qOWW/kVx46PjU+JC+tYwfmn+LEMMoB5WwPI4CzhGRM4E6YACOJTJIRKpd62MssMqd3wiMAxpFpBoYCGwMP1RVbwVuBZg5c2YXh0qNSpGrSNBPRyi43vVuq4ixGMunw18kmIxvwf6zi6bz78fsWdDnNIzuQNktD1W9RlXHqupE4CLgKVX9JPA08DF32qXAQ+7xw+457vWntJit1IweSc72JL7L7aGAeVevx1GWjH+s1Rf06Egqm9x9vne0dTB72abY5x4wZmCqrblh9BS6U5HgfwFfFZElODGN293x24Gh7vhXgasrJJ/RCWYv21jU9qk525P4jjNiHmVwW/k/07qt6TjF//e/s1mweisAb67cwmNvr8m497PH7dWl8hlGOalokaCqPgM84x6/DxwWMWcncEFZBTO6lCcXruXyO2fz/XP3z1l/EKYS2VaqSlRCX2Sdh+8dC11lAfBek9PM8IQpw3n63WAM7roP78enjphAjTUdNHow9tdrlJwV7p7WS9ZtL/jeSmRbxd0WdosBrPVZG798ekmmfAQtjITAZUdNMsVh9HisPYlRNopZynO7rXzZVhkxj+KUR0dSUzvvZXueqrJ0faZCPH/6GB58fSXg1GHsO3oAnztuLwbUV0daNIbRE7GvP0a3JqcCKInlEX1f+Hmrtuzk/tmNgbHLj57ETy84GHD2/t539AAABjbUmOIwehVmeRhlo5ilM8pV5Me/0HdVnUeU8vje3xZwx4tLI+dfdtRENre08Z+nTWGPgXWICK9962QG1Nv/Xkbvxf66jbJRzFKey3rwK4xwnUexyVb+d6oqv3hqSaziOHaf4Vz34f0zxof371Pcyw2jh2BuK6Nbk8t68Fsm/t5SC1dv5SePv1vUO/06aOvOdm6ctSh27ghTEsZuiikPo1sTtibCxFkeX7zn9YLfNaDOMcRXbt7BG42b2dzSyvf/viBj3mNfPobPfGgiAHsN71fwewyjN2BuK6Nb0xGjOx6at5L7Z68g4QtC+xVJtk60cey/x0Been8DZ978PABnHzSav7+xOmPe1FED+M45+3P6AaOYOWFwwe8xjN6AKQ+jW+O3JrxqbhHhu39bwMbmYFvzRWu38+TCtZy070j69cn80z5p6giefGdd7LvC6bl+xbHX8L58/bSpNNSmd947Ys+hhX0Yw+hFmNvKKDk5Eqay4rcmvnLfPD5xm9ORtqYqM3frCbeS/YMNzbyyNKN3JidMHRH5jk8e7nSw7dsnekvWfUcP4ImvHsfpB4zi2H2GF/wZDKM3YsrDKDltcb6nPPBnPv113ipeen8Dzy9uYu3WXfStjV7sj/vJM4Hzh686ipsuPJgPH7RH5Pwrjt2TE6eO4Fo3a+rjM8dy2MQhAPzHMZP4+xeOthoNwwhhbiuj5LR3kfLw8Arzjp86gkciYhJ+Fv3gDGqrExw0dhBbd7ZlXP/xRw9kwtC+3PGZQwFY8L3T6FNdxZsrt3Der17k3GljIqvNDWN3xywPo+S0un6rYtqFtLR2ZIz9bf4q6muqMmIOh4SC11UJCQTO+9VWc/yU4fz+skNTY/5Nl8DZba8qIUwbN4hlN5zFAWMGFiyzYewOmOVhlBzPbZWrWjyK2cvSsYuDxw1i/orNAPz8omls2RG0JG76+DSO/cnTXPfh/Zg4tC9jB9cHricSwh8uy2jcbBhGEZjyMEqO57ZqzdN91daRZOuONpq27wrsg3Hbpw/hsOufBODU/UexYNXWwH3jhzYw59snM6RvrcUoDKPEmPIwSk6ba3Hksjzmr9jMR2/5VyrDasLQBpIKv73kEN5vamZ4vz78/QtHU+1mWu23xwDmX3sqW3a0Ue8Gz4f2y6/i++aLp9O/zv78DaNY7P8eo+R4bquorKsN23cxb8VmTtp3JI++tTqQmvvBhhZ+duE0Ttt/VGosHIMY2FDDwIaagmU65+DozCvDMPKj7MpDRMYBdwGjgCRwq6r+XESGAPcBE4FlwMdVdZM4/oefA2cCLcBnVHVuueU2isevPLxGgw21VZx90B4c8SPHDdWvjxOonjF+EJt3tPG+uxPf8VOsrsIwuiOVyLZqB76mqvsCRwBXish+OHuTP6mqk4EnSe9VfgYw2f25Aril/CIbncFzWz2xcB1L1zdz46xF/OCRhSnFAbB9VztbdrRx+J5Duf3SdDbUoIbasstrGEZuyq48VHW1Zzmo6jZgITAGOBe40512J3Cee3wucJc6vAwMEpHRZRbbKILW9iTXPvQWr/qqvR+cuzJy7rB+jpI4aq9hjB/SwCVHTODRLx5TFjkNwyicisY8RGQiMB14BRipqqvBUTAi4vWSGAOs8N3W6I4FqsNE5Aocy4Tx44O5+0b5UVX+MreRu176AICRA/qwduuu1D7fnzt+L2qrEvz8ycUA/PMrx7Fq8w7232MAIsL3zzugYrIbhpGbiikPEekH/AX4sqpuzZJaGXUhI21HVW8FbgWYOXNmJ7opGZ1FVTn6x0+zcvOO1NhBYwcxa8FaAPrWVvG1U/ahuirBqIF17Dd6AEP61jKkr7moDKOnUBHlISI1OIrjT6r6oDu8VkRGu1bHaMBrf9oIjPPdPhZYVT5pjVxsbG7llmeWcO60MUwa1pev/3l+QHGA48L6+mlTGFBfw3GTh1Nd5XhMLz7MrETD6IlUIttKgNuBhap6o+/Sw8ClwA3u74d841eJyL3A4cAWz71lVJa2jiQ3/OMd5nywiXkrNnPb85lbtfavq2bbznbaOpJcecLeFZDSMIxSUAnL4yjgEuBNEZnnjn0TR2ncLyKXA8uBC9xrj+Kk6S7BSdW9rLziGmE2t7RyzI+fZsSAPrznptSGufTICdz50gd8dMZYNjS38qWTTHEYRm+i7MpDVV8gOo4BcFLEfAWuLKlQRiTtHUl+8dQS5i7fxNdOncI+I/uxdH0zZ938AgDbmtoj7/vojLF8+kMTWb+9la+csg8D6wsv4jMMo3tjFeZGAFWlubWDVZt38NX75/HWSqd/1POL10fO//LJk/nZE4tT57decggn7TuSqoTwq0/OKIvMhmGUH1MeBgBL1zczbnA9d7y4lB8++k5e9/zo/AO5+LDxfPnkfXh3zTZ+++x7nDB1hO1/YRi7AaY8dkN2tnXw73fO5qoT9+aIPYfyryXr+cTvXomc+8OPHMiJU0fwi6cW86dXlnPwuEFccMhY9hhUx4lTR6bmTRnVnxsvnFauj2AYRoUx5dGLueruuUwe0Z8vnTw5MD73g028sGQ9LyxZz8n7juSJhWsz7v35RdOYMX4wYwfXIyJc7yqRQyYMtpYhhmHYToK9lWRS+fsbq7npiUWB8faOJA/MaUyde4rj/BljAvPOnTaGcUMaAvtinLTvSFMchmEAZnkUzUPzVpJU5SPTx1ZalAx2tHbwhXvSjYfXb9/Fhb99ifNnjOXe15azYuMO+vWp5rZPz+TKu+eysbmV86eP5d+OmsQdLy7lxKkjsjzdMAwDRIvYV7q7M3PmTJ09e3ZJ3zHx6kcAWHbDWSV9TzZO/9lznHngaC48dBy3Pfc+XzhxMgMbarjrpWVc+9DbWe89ZMJg/vK5D7GjtYNXlm7g+JKfS/kAAAqKSURBVCmmMAxjd0dE5qjqzHzmmuXRw1i4eisLVm1l84423lmzjXfWbOPGWY5r6ncvLOWYycN4fvF6+lQn+O45+3P1g28G7j980hA+86GJTB8/GID62ipTHIZhFIwpj27M2q07qauuoi2ZZJi7verHbvkXza0dsfc8v3g9B4wZwO2XHsrIAXXU1VTx7KImrj17P373wvt89ri96F9nRXuGYXQOUx5FkPRtldrekUw1+SuGJxasRYEXl6xn3JAGbnnmPT6011AOnTSE//7rW6l5Zx44im072wOKY2jfWm746EH061NN0/ZdrNjYQltHkgtmjmPkgDoAzps+hvOmO8Hwr582tWg5DcMw/JjyKILtrem2HFt3tqdaia/dujO1aHuoKvNWbObJhetIJITZyzayavMOdrYlGdqvlrdXbc14/sPzV/Hw/GDj4EffXAPAV0/ZhyuO3ZOECNUJIWEFeYZhVABTHgXSvKud15dvTp3PXraRzTvaeGjeSl5csoHRA+s4/YBR9Kmu4q+vr2TN1p2B+8cMqmf0wDpmf7Ap49rJ+45g4eptrNy8g4lDG/j8CXuzdstOxgyu56v3z+f4KcP54knBmg3DMIxKYNlWIf713nr2Gz2ABau2snDNNob0reGDDS384V/L2L6znfZk/v9ex0wextL1zYjAtWfvDzgKQkRo60jSkVRqqxJ8669vst8eA7nkiAk8NG8lX7p3HjddeHAgDXjbzjZqqxP0qa4q6nMZhmHkwrKtimT5hhY+cVt0m44xg+qZOqo/dTVVvL58Mx+dMZZzp+3BknXbGTWwjgWrtrJ43TY2t7Rx+dGTqEoIh0wYTFIhIRDeKbGmKkGNqwd+dP5BqfFzDt6DPQbVM3PC4MB8C3IbhtGdMMvDRzKpPLu4iecWNXHUXsPYZ2R//vbGKkYPrOMj08dkKADDMIzeRCGWhykPwzAMAyhMefSY3lYicrqIvCsiS0Tk6krLYxiGsTvTI5SHiFQBvwLOAPYDLhaR/SorlWEYxu5Lj1AewGHAElV9X1VbgXuBcyssk2EYxm5LT1EeY4AVvvNGd8wwDMOoAD1FeUSlOQUi/SJyhYjMFpHZTU1NZRLLMAxj96SnKI9GYJzvfCwQ6N+hqreq6kxVnTl8+PCyCmcYhrG70VOUx2vAZBGZJCK1wEXAwxWWyTAMY7elR1SYq2q7iFwFPA5UAXeoavbdjgzDMIyS0SuLBEWkCfigyNuHAeu7UJxyYDKXB5O5PJjM5SFK5gmqmpffv1cqj84gIrPzrbDsLpjM5cFkLg8mc3norMw9JeZhGIZhdCNMeRiGYRgFY8ojk1srLUARmMzlwWQuDyZzeeiUzBbzMAzDMArGLA/DMAyjYEx5+Oiubd9F5A4RWScib/nGhojILBFZ7P4e7I6LiNzsfoY3RGRGBeQdJyJPi8hCEXlbRL7UA2SuE5FXRWS+K/N33fFJIvKKK/N9bpEqItLHPV/iXp9Ybpl9sleJyOsi8veeILOILBORN0VknojMdse67d+GK8cgEfmziLzj/l0f2Z1lFpEp7r+v97NVRL7cpTKrqv04rrsq4D1gT6AWmA/sV2m5XNmOBWYAb/nG/ge42j2+Gvixe3wm8A+cfmBHAK9UQN7RwAz3uD+wCKeVfneWWYB+7nEN8Iory/3ARe74b4DPucefB37jHl8E3FfBv4+vAncDf3fPu7XMwDJgWGis2/5tuHLcCfy7e1wLDOruMvtkrwLWABO6UuaKfaDu9gMcCTzuO78GuKbScvnkmRhSHu8Co93j0cC77vFvgYuj5lVQ9oeAU3qKzEADMBc4HKeIqjr8N4LT7eBI97janScVkHUs8CRwIvB393/+7i5zlPLotn8bwABgafjfqjvLHJLzVODFrpbZ3FZpelrb95GquhrA/T3CHe9Wn8N1jUzH+SbfrWV23T/zgHXALBxLdLOqtkfIlZLZvb4FGFpeiQH4GfANIOmeD6X7y6zAP0Vkjohc4Y5157+NPYEm4Peue/B3ItKX7i2zn4uAe9zjLpPZlEeanG3fewjd5nOISD/gL8CXVXVrtqkRY2WXWVU7VHUazrf5w4B9o6a5vysus4icDaxT1Tn+4Yip3UZml6NUdQbOzqBXisixWeZ2B5mrcdzGt6jqdKAZx+UTR3eQGQA33nUO8ECuqRFjWWU25ZEmZ9v3bsZaERkN4P5e5453i88hIjU4iuNPqvqgO9ytZfZQ1c3AMzi+30Ei4jUQ9cuVktm9PhDYWF5JOQo4R0SW4eyueSKOJdKdZUZVV7m/1wH/h6Oou/PfRiPQqKqvuOd/xlEm3VlmjzOAuaq61j3vMplNeaTpaW3fHwYudY8vxYkreOOfdrMnjgC2eGZquRARAW4HFqrqjb5L3Vnm4SIyyD2uB04GFgJPAx+Lkdn7LB8DnlLXWVwuVPUaVR2rqhNx/l6fUtVP0o1lFpG+ItLfO8bxx79FN/7bUNU1wAoRmeIOnQQs6M4y+7iYtMsKulLmSgVxuuMPTsbBIhxf97cqLY9PrnuA1UAbzjeEy3F81U8Ci93fQ9y5AvzK/QxvAjMrIO/ROCbvG8A89+fMbi7zQcDrrsxvAde643sCrwJLcEz/Pu54nXu+xL2+Z4X/Ro4nnW3VbWV2ZZvv/rzt/X/Wnf82XDmmAbPdv4+/AoN7gMwNwAZgoG+sy2S2CnPDMAyjYMxtZRiGYRSMKQ/DMAyjYEx5GIZhGAVjysMwDMMoGFMehmEYRsGY8jCMECLSEepImrXDsoh8VkQ+3QXvXSYiw4q47zQR+Y6IDBaRRzsrh2HkQ3XuKYax27FDnTYleaGqvymlMHlwDE5h4LHAixWWxdhNMOVhGHnitgG5DzjBHfqEqi4Rke8A21X1pyLyReCzQDuwQFUvEpEhwB04BXItwBWq+oaIDMUpAB2OU7Qnvnd9CvgiTvvvV4DPq2pHSJ4Lcbo/7wmcC4wEtorI4ap6Tin+DQzDw9xWhpFJfchtdaHv2lZVPQz4JU4fqTBXA9NV9SAcJQLwXeB1d+ybwF3u+HXAC+o023sYGA8gIvsCF+I0EJwGdACfDL9IVe8jvc/LgTiV8dNNcRjlwCwPw8gkm9vqHt/vmyKuvwH8SUT+itPGApx2LR8FUNWnRGSoiAzEcTOd744/IiKb3PknAYcArzltwqgn3cAuzGSclhIADaq6LY/PZxidxpSHYRSGxhx7nIWjFM4B/ltE9id7u+uoZwhwp6pek00QcbZwHQZUi8gCYLS7H8kXVPX57B/DMDqHua0MozAu9P1+yX9BRBLAOFV9GmeDpkFAP+A5XLeTiBwPrFdnfxP/+Bk4zfbAaVj3MREZ4V4bIiITwoKo6kzgEZx4x//gNBmcZorDKAdmeRhGJvXuN3iPx1TVS9ftIyKv4Hzxujh0XxXwR9clJcBNqrrZDaj/XkTewAmYey2xvwvcIyJzgWeB5QCqukBEvo2z214Cp5vylcAHEbLOwAmsfx64MeK6YZQE66prGHniZlvNVNX1lZbFMCqNua0MwzCMgjHLwzAMwygYszwMwzCMgjHlYRiGYRSMKQ/DMAyjYEx5GIZhGAVjysMwDMMoGFMehmEYRsH8/9oevyJUElDlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x13bc3ef0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
